Skip to main content

Output Specification

Overview

The output of the DeepTone processing functions is json. When processing a stream, the results are returned one time step at a time. When processing a file, all time steps are returned, together with some additional file-level summaries. Additionally, in both cases raw model outputs can be included in the result.

In this section we present the different components of the DeepTone output:

Single time step, single model

{
"timestamp":1024,
"results":{
"gender":{
"result": "no_speech",
"confidence": 0.7088
}
}
}

Single time step, multiple models

{
"timestamp":2048,
"results":{
"arousal":{
"result": "no_speech",
"confidence": 0.7597
},
"gender":{
"result": "no_speech",
"confidence": 0.7597
}
}
}

Single time step, single model, raw values included

{
"timestamp":1024,
"results":{
"gender":{
"result": "no_speech",
"confidence": 0.7088
}
},
"raw": {
"gender":{
"male": 0.1211,
"female": 0.8789
}
}
}

File-processing output

There are several additional components in the file processing output compared to the streaming output.

The additional functionalities listed bellow correspond to additional keys in the output:

  • processing multiple audio channels - channels
  • processing and delivering results for all time steps - time_series
  • calculating file-level summary - summary
  • calculating file-level transitions - transitions
  • including raw model outputs - raw_outputs

Those differences are reflected in the different keys of the json output described in the following sections.

You can also head straight to a complete example.

channels

When processing files, the high-level structure of the output is shown below. The top level key channels holds the data for each channel - 0, 1 or both.

For each channel, the output contains time_series, and optionally summary and transitions.

{
"channels":{
"0":{
"time_series":[],
"summary":{},
"transitions":{}
}
}
}

time_series

The time_series key holds an array of results, under the results key, one for each time step as defined in the sections above. Optionally, it also holds an array of raw results under raw.

The format is:

{
"channels":{
<CHANNEL_IDENTIFIER>:{
"time_series":[
{
"timestamp":<START_TS_OF_STEP>,
"results": {
<MODEL_NAME>:{
"result":<MODEL_RESULT>,
"confidence":<MODEL_CONFIDENCE>
}
}
},
{
"timestamp":<START_TS_OF_STEP>,
"results": {
<MODEL_NAME>:{
"result"::<MODEL_RESULT>,
"confidence":<MODEL_CONFIDENCE>
}
}
},
...
],
"summary":{...},
"transitions":{...}
}
}
}

An example with real values from the Arousal model:

{
"channels":{
"0":{
"time_series":[
{
"timestamp":0,
"results": {
"arousal":{
"result":"no_speech",
"confidence":0.748
}
}
},
{
"timestamp":4096,
"results": {
"arousal":{
"result":"no_speech",
"confidence":0.7105
}
}
},
{
"timestamp":8192,
"results":{
"arousal":{
"result":"high",
"confidence":0.9926
}
}
},
...
],
"summary":{...},
"transitions":{...}
}
}
}

time_series with multiple models

In case multiple models are used, this will be reflected in the result object for each time step, as explained the multiple model time step result.

For example with speech and arousal, one should expect a key corresponding to each model's name in the results object for a single time step:

{
"channels":{
"0":{
"time_series":[
{
"timestamp":0,
"results":{
"speech":{
"result":"speech",
"confidence":0.748
},
"arousal":{
"result":"neutral",
"confidence":0.658
}
}
},
...
],
"summary":{...},
"transitions":{...}
}
}
}

time_series with raw model outputs

When you request raw values, an additional raw key will be included for each time step of the results. It will hold a dictionary of raw model outputs for every requested model, with all class names as keys and their respective probabilities as values.

The format is:

{
"channels":{
<CHANNEL_IDENTIFIER>:{
"time_series":[
{
"timestamp":<START_TS_OF_STEP>,
"results":{
<MODEL_NAME>:{
"result":<MODEL_RESULT>,
"confidence":<MODEL_CONFIDENCE>
}
},
"raw": {
<MODEL_NAME>:{
"<CLASS1>": <CLASS1_PROBABILITY>,
"<CLASS2>": <CLASS2_PROBABILITY>,
"<CLASS3>": <CLASS3_PROBABILITY>
}
}
},
{
"timestamp":<START_TS_OF_STEP>,
"results":{
<MODEL_NAME>:{
"result"::<MODEL_RESULT>,
"confidence":<MODEL_CONFIDENCE>
}
},
"raw": {
<MODEL_NAME>:{
"<CLASS1>": <CLASS1_PROBABILITY>,
"<CLASS2>": <CLASS2_PROBABILITY>,
"<CLASS3>": <CLASS3_PROBABILITY>
}
}
},
...
],
"summary":{...},
"transitions":{...}
}
}
}

An example with real values from the Arousal model:

{
"channels":{
"0":{
"time_series":[
{
"timestamp":0,
"results":{
"arousal":{
"result":"no_speech",
"confidence":0.748
}
},
"raw": {
"arousal": {
"high": 0.1143,
"low": 0.0034,
"neutral": 0.8823
}
}
},
{
"timestamp":4096,
"results":{
"arousal":{
"result":"no_speech",
"confidence":0.7105
}
},
"raw": {
"arousal": {
"high": 0.0078,
"low": 0.2143,
"neutral": 0.7779
}
}
},
{
"timestamp":8192,
"results":{
"arousal":{
"result":"high",
"confidence":0.9926
}
},
"raw": {
"arousal": {
"high": 0.9926,
"low": 0.0071,
"neutral": 0.0003
}
}
},
...
],
"summary":{...},
"transitions":{...}
}
}
}

summary

The summary key holds file-level calculations which summarise the prevalence of each class for a corresponding model.

{
"channels":{
<CHANNEL_IDENTIFIER>:{
"time_series":[],
"summary":{
<MODEL_NAME>:{
"<CLASS1>_fraction":0.5416,
"<CLASS2>_fraction":0.2609,
"<CLASS3>_fraction":0.1975
}
},
"transitions":{...}
}
}
}

An example with both the arousal and speech models:

{
"channels":{
"0":{
"time_series":[],
"summary":{
"speech":{
"music_fraction":0.5416,
"other_fraction":0.2609,
"speech_fraction":0.1975,
"silence_fraction": 0.0
},
"arousal":{
"high_fraction":0.1793,
"low_fraction":0.0071,
"neutral_fraction":0.0111,
"no_speech_fraction":0.8025,
"silence_fraction": 0.0
}
},
"transitions":{...}
}
}
}

transitions

The transitions key represents a summary of the timeseries. The idea is to indicate the starting time and ending time of each segment of audio, effectively collapsing time steps with the same model output. For more details on interpretation see the description of transitions in the corresponding model section.

Below is the output format, with an array of transitions for each model. Each element in the array represent an uninterrupted segment of the audio classified with the same value - e.g. music between from timestamp_start to timestamp_end.

{
"channels":{
<CHANNEL_IDENTIFIER>:{
"time_series":[],
"summary":{},
"transitions":{
<MODEL_NAME>:[
{
"timestamp_start":<START_TS_OF_SEGMENT>,
"timestamp_end":<END_TS_OF_SEGMENT>,
"result":<MODEL_RESULT_FOR_SEGMENT>,
"confidence":<MODEL_CONFIDENCE>
},
{
"timestamp_start":<START_TS_OF_SEGMENT>,
"timestamp_end":<END_TS_OF_SEGMENT>,
"result":<MODEL_RESULT_FOR_SEGMENT>,
"confidence":<MODEL_CONFIDENCE>
},
...
]
}
}
}

In the example with real data, you can see a music segment, followed by a speech segment.

{
"channels":{
"0":{
"time_series":[],
"summary":{},
"transitions":{
"speech":[
{
"timestamp_start":0,
"timestamp_end":6144,
"result":"music",
"confidence":0.7751
},
{
"timestamp_start":6144,
"timestamp_end":8128,
"result":"speech",
"confidence":0.5445
},
...
]
}
}
}

Complete example

Below you can find a complete example with speech and arousal models. You can paste it in https://jsonformatter.curiousconcept.com/ to explore it more easily.

The function call used to generate it is:

output = engine.process_file(
filename="vlog_2.wav",
models=[engine.models.Speech, engine.models.Arousal],
output_period=4096,
channel=0,
use_chunking=True,
include_summary=True,
include_transitions=True,
include_raw_values=True,
volume_threshold=0
)

Complete output:

{
"channels":{
"0":{
"time_series":[
{
"timestamp": 0,
"results":{
"speech": {
"result": "other",
"confidence": 0.6094
},
"arousal": {
"result": "no_speech",
"confidence": 0.6094
}
},
"raw": {
"speech": {
"music": 0.1735,
"other": 0.6094,
"speech": 0.2171
},
"arousal": {
"high": 0.2926,
"low": 0.1071,
"neutral": 0.6003
}
}
},
{
"timestamp": 4096,
"results":{
"speech": {
"result": "other",
"confidence": 0.6441
},
"arousal": {
"result": "no_speech",
"confidence": 0.6441
}
},
"raw": {
"speech": {
"music": 0.1357,
"other": 0.6441,
"speech": 0.2202
},
"arousal": {
"high": 0.3461,
"low": 0.201,
"neutral": 0.4529
}
}
},
{
"timestamp": 8192,
"results":{
"speech": {
"result": "speech",
"confidence": 0.5952
},
"arousal": {
"result": "high",
"confidence": 0.9992
}
},
"raw": {
"speech": {
"music": 0.1034,
"other": 0.3014,
"speech": 0.5952
},
"arousal": {
"high": 0.9992,
"low": 0.0002,
"neutral": 0.0006
}
}
}
],
"summary": {
"speech": {
"music_fraction": 0.0,
"other_fraction": 0.6543,
"speech_fraction": 0.3457,
"silence_fraction": 0.0
},
"arousal": {
"high_fraction": 0.3148,
"low_fraction": 0.0185,
"neutral_fraction": 0.0123,
"no_speech_fraction": 0.6543,
"silence_fraction": 0.0
}
},
"transitions": {
"speech": [
{
"timestamp_start": 0,
"timestamp_end": 64,
"result": "other",
"confidence": 0.779
},
{
"timestamp_start": 64,
"timestamp_end": 128,
"result": "speech",
"confidence": 0.6974
},
{
"timestamp_start": 128,
"timestamp_end": 448,
"result": "other",
"confidence": 0.5428
},
{
"timestamp_start": 448,
"timestamp_end": 512,
"result": "speech",
"confidence": 0.7051
},
{
"timestamp_start": 512,
"timestamp_end": 704,
"result": "other",
"confidence": 0.6106
},
{
"timestamp_start": 704,
"timestamp_end": 1152,
"result": "speech",
"confidence": 0.6448
},
{
"timestamp_start": 1152,
"timestamp_end": 1280,
"result": "other",
"confidence": 0.5515
},
{
"timestamp_start": 1280,
"timestamp_end": 1344,
"result": "speech",
"confidence": 0.5107
},
{
"timestamp_start": 1344,
"timestamp_end": 2816,
"result": "other",
"confidence": 0.7687
},
{
"timestamp_start": 2816,
"timestamp_end": 2880,
"result": "speech",
"confidence": 0.4946
},
{
"timestamp_start": 2880,
"timestamp_end": 2944,
"result": "other",
"confidence": 0.6252
},
{
"timestamp_start": 2944,
"timestamp_end": 3072,
"result": "speech",
"confidence": 0.6283
},
{
"timestamp_start": 3072,
"timestamp_end": 3200,
"result": "other",
"confidence": 0.572
},
{
"timestamp_start": 3200,
"timestamp_end": 3648,
"result": "speech",
"confidence": 0.6887
},
{
"timestamp_start": 3648,
"timestamp_end": 4864,
"result": "other",
"confidence": 0.8488
},
{
"timestamp_start": 4864,
"timestamp_end": 4992,
"result": "speech",
"confidence": 0.5812
},
{
"timestamp_start": 4992,
"timestamp_end": 5760,
"result": "other",
"confidence": 0.6926
},
{
"timestamp_start": 5760,
"timestamp_end": 5824,
"result": "speech",
"confidence": 0.5013
},
{
"timestamp_start": 5824,
"timestamp_end": 6144,
"result": "other",
"confidence": 0.5425
},
{
"timestamp_start": 6144,
"timestamp_end": 6272,
"result": "speech",
"confidence": 0.506
},
{
"timestamp_start": 6272,
"timestamp_end": 6464,
"result": "other",
"confidence": 0.5957
},
{
"timestamp_start": 6464,
"timestamp_end": 6656,
"result": "speech",
"confidence": 0.5601
},
{
"timestamp_start": 6656,
"timestamp_end": 7936,
"result": "other",
"confidence": 0.6909
},
{
"timestamp_start": 7936,
"timestamp_end": 8512,
"result": "speech",
"confidence": 0.6191
},
{
"timestamp_start": 8512,
"timestamp_end": 8576,
"result": "other",
"confidence": 0.5203
},
{
"timestamp_start": 8576,
"timestamp_end": 8768,
"result": "speech",
"confidence": 0.5293
},
{
"timestamp_start": 8768,
"timestamp_end": 8896,
"result": "other",
"confidence": 0.5398
},
{
"timestamp_start": 8896,
"timestamp_end": 8960,
"result": "speech",
"confidence": 0.4743
},
{
"timestamp_start": 8960,
"timestamp_end": 9408,
"result": "other",
"confidence": 0.6112
},
{
"timestamp_start": 9408,
"timestamp_end": 10368,
"result": "speech",
"confidence": 0.7707
}
],
"arousal": [
{
"timestamp_start": 0,
"timestamp_end": 64,
"result": "no_speech",
"confidence": 0.779
},
{
"timestamp_start": 64,
"timestamp_end": 128,
"result": "high",
"confidence": 0.492
},
{
"timestamp_start": 128,
"timestamp_end": 448,
"result": "no_speech",
"confidence": 0.5428
},
{
"timestamp_start": 448,
"timestamp_end": 512,
"result": "high",
"confidence": 0.8254
},
{
"timestamp_start": 512,
"timestamp_end": 704,
"result": "no_speech",
"confidence": 0.6106
},
{
"timestamp_start": 704,
"timestamp_end": 1152,
"result": "high",
"confidence": 0.9887
},
{
"timestamp_start": 1152,
"timestamp_end": 1280,
"result": "no_speech",
"confidence": 0.5515
},
{
"timestamp_start": 1280,
"timestamp_end": 1344,
"result": "high",
"confidence": 0.9965
},
{
"timestamp_start": 1344,
"timestamp_end": 2816,
"result": "no_speech",
"confidence": 0.7687
},
{
"timestamp_start": 2816,
"timestamp_end": 2880,
"result": "low",
"confidence": 0.8157
},
{
"timestamp_start": 2880,
"timestamp_end": 2944,
"result": "no_speech",
"confidence": 0.6252
},
{
"timestamp_start": 2944,
"timestamp_end": 3072,
"result": "low",
"confidence": 0.6866
},
{
"timestamp_start": 3072,
"timestamp_end": 3200,
"result": "no_speech",
"confidence": 0.572
},
{
"timestamp_start": 3200,
"timestamp_end": 3648,
"result": "high",
"confidence": 0.8684
},
{
"timestamp_start": 3648,
"timestamp_end": 4864,
"result": "no_speech",
"confidence": 0.8488
},
{
"timestamp_start": 4864,
"timestamp_end": 4992,
"result": "neutral",
"confidence": 0.7727
},
{
"timestamp_start": 4992,
"timestamp_end": 5760,
"result": "no_speech",
"confidence": 0.6926
},
{
"timestamp_start": 5760,
"timestamp_end": 5824,
"result": "high",
"confidence": 0.9167
},
{
"timestamp_start": 5824,
"timestamp_end": 6144,
"result": "no_speech",
"confidence": 0.5425
},
{
"timestamp_start": 6144,
"timestamp_end": 6272,
"result":" high",
"confidence": 0.9392
},
{
"timestamp_start": 6272,
"timestamp_end": 6464,
"result": "no_speech",
"confidence": 0.5957
},
{
"timestamp_start": 6464,
"timestamp_end": 6656,
"result": "high",
"confidence": 0.9931
},
{
"timestamp_start": 6656,
"timestamp_end": 7936,
"result": "no_speech",
"confidence": 0.6909
},
{
"timestamp_start": 7936,
"timestamp_end": 8512,
"result": "high",
"confidence": 0.9999
},
{
"timestamp_start": 8512,
"timestamp_end": 8576,
"result": "no_speech",
"confidence": 0.5203
},
{
"timestamp_start": 8576,
"timestamp_end": 8768,
"result":" high",
"confidence": 0.9992
},
{
"timestamp_start": 8768,
"timestamp_end": 8896,
"result": "no_speech",
"confidence": 0.5398
},
{
"timestamp_start": 8896,
"timestamp_end": 8960,
"result": "high",
"confidence": 1.0
},
{
"timestamp_start": 8960,
"timestamp_end": 9408,
"result": "no_speech",
"confidence": 0.6112
},
{
"timestamp_start": 9408,
"timestamp_end": 10368,
"result": "high",
"confidence": 0.999
}
]
}
}
}
}
  • Overview
  • Single time step, single model
  • Single time step, multiple models
  • Single time step, single model, raw values included
  • File-processing output
    • channels
    • time_series
    • time_series with multiple models
    • time_series with raw model outputs
    • summary
    • transitions
  • Complete example