Skip to main content

Batch Processing

DeepTone™'s File Processing functionality allows you to process large numbers of files as fast as possible - also called batch processing.

Before reading this page, you should already be familiar with the file processing functionality and its limitations.

There are a few things to take into consideration when processing batches of files:

  1. It is important to use the URL Method, if the batch size is greater than 10 files. Otherwise the Direct File Upload Method can be used.
  2. When using the Direct File Upload Method make sure to send requests sequentially.
  3. To reduce the number of requests that are needed to check if a processing job is finished, it is recommended to use the callbacks functionality. If this is not an option for you, it makes sense to introduce a backoff into the job status checks. For example: Instead of checking the status of a job every second, check the status of a job every n seconds, where n is the number of times this job's status has already been checked.
  4. It is possible that a processing job fails sometimes. It makes sense to retry failed jobs with an exponential backoff for a maximum number of 3 to 5 times. If a processing job fails 5 times or more and it is not clear why, please report to support@oto.ai, so we can look into it.

A simple example of a batch processing Python script can be found here . The script will save all created job objects from a given list of urls and their results as files.