DeepTone™'s File Processing functionality allows you to process large numbers of files as fast as possible - also called batch processing.
Before reading this page, you should already be familiar with the file processing functionality and its limitations.
There are a few things to take into consideration when processing batches of files:
- It is important to use the URL Method, if the batch size is greater than 10 files. Otherwise the Direct File Upload Method can be used.
- When using the Direct File Upload Method make sure to send requests sequentially.
- To reduce the number of requests that are needed to check if a processing job is finished, it is recommended to use
the callbacks functionality. If this is not an option for you,
it makes sense to introduce a backoff into the job status checks. For example: Instead of checking the status of a job
every second, check the status of a job every
nseconds, where n is the number of times this job's status has already been checked.
- It is possible that a processing job fails sometimes. It makes sense to retry failed jobs with an exponential backoff for a maximum number of 3 to 5 times. If a processing job fails 5 times or more and it is not clear why, please report to firstname.lastname@example.org, so we can look into it.
A simple example of a batch processing Python script can be found here . The script will save all created job objects from a given list of urls and their results as files.