Output Specification
Overview
The output of the DeepTone processing functions is json. When processing a stream, the results are returned one time step at a time. When processing a file, all time steps are returned, together with some additional file-level summaries. Additionally, in both cases raw model outputs can be included in the result.
In this section we present the different components of the DeepTone output:
- json output for a single time step and single model - real-time case
- json output for a single time step and multiple models - real-time case
- json output for a single time step and single model returning raw values - real-time case
- json output for multiple time steps - file-processing case
Single time step, single model
{
"timestamp":1024,
"results":{
"gender":{
"result": "no_speech",
"confidence": 0.7088
}
}
}
Single time step, multiple models
{
"timestamp":2048,
"results":{
"arousal":{
"result": "no_speech",
"confidence": 0.7597
},
"gender":{
"result": "no_speech",
"confidence": 0.7597
}
}
}
Single time step, single model, raw values included
{
"timestamp":1024,
"results":{
"gender":{
"result": "no_speech",
"confidence": 0.7088
}
},
"raw": {
"gender":{
"male": 0.1211,
"female": 0.8789
}
}
}
File-processing output
There are several additional components in the file processing output compared to the streaming output.
The additional functionalities listed bellow correspond to additional keys in the output:
- processing multiple audio channels -
channels - processing and delivering results for all time steps -
time_series - calculating file-level
summary-summary - calculating file-level
transitions-transitions - including
rawmodel outputs -raw_outputs
Those differences are reflected in the different keys of the json output described in the following sections.
You can also head straight to a complete example.
channels
When processing files, the high-level structure of the output is shown below. The top level key channels holds the data for each channel - 0, 1 or both.
For each channel, the output contains time_series, and optionally summary and transitions.
{
"channels":{
"0":{
"time_series":[],
"summary":{},
"transitions":{}
}
}
}
time_series
The time_series key holds an array of results, under the results key, one for each time step as defined in the sections above. Optionally, it also holds an array of raw results under raw.
The format is:
{
"channels":{
<CHANNEL_IDENTIFIER>:{
"time_series":[
{
"timestamp":<START_TS_OF_STEP>,
"results": {
<MODEL_NAME>:{
"result":<MODEL_RESULT>,
"confidence":<MODEL_CONFIDENCE>
}
}
},
{
"timestamp":<START_TS_OF_STEP>,
"results": {
<MODEL_NAME>:{
"result"::<MODEL_RESULT>,
"confidence":<MODEL_CONFIDENCE>
}
}
},
...
],
"summary":{...},
"transitions":{...}
}
}
}
An example with real values from the Arousal model:
{
"channels":{
"0":{
"time_series":[
{
"timestamp":0,
"results": {
"arousal":{
"result":"no_speech",
"confidence":0.748
}
}
},
{
"timestamp":4096,
"results": {
"arousal":{
"result":"no_speech",
"confidence":0.7105
}
}
},
{
"timestamp":8192,
"results":{
"arousal":{
"result":"high",
"confidence":0.9926
}
}
},
...
],
"summary":{...},
"transitions":{...}
}
}
}
time_series with multiple models
In case multiple models are used, this will be reflected in the result object for each time step, as explained the multiple model time step result.
For example with speech and arousal, one should expect a key corresponding to each model's name in the results object for a single time step:
{
"channels":{
"0":{
"time_series":[
{
"timestamp":0,
"results":{
"speech":{
"result":"speech",
"confidence":0.748
},
"arousal":{
"result":"neutral",
"confidence":0.658
}
}
},
...
],
"summary":{...},
"transitions":{...}
}
}
}
time_series with raw model outputs
When you request raw values, an additional raw key will be included for each time step of the results. It will hold a dictionary of raw model outputs
for every requested model, with all class names as keys and their respective probabilities as values.
The format is:
{
"channels":{
<CHANNEL_IDENTIFIER>:{
"time_series":[
{
"timestamp":<START_TS_OF_STEP>,
"results":{
<MODEL_NAME>:{
"result":<MODEL_RESULT>,
"confidence":<MODEL_CONFIDENCE>
}
},
"raw": {
<MODEL_NAME>:{
"<CLASS1>": <CLASS1_PROBABILITY>,
"<CLASS2>": <CLASS2_PROBABILITY>,
"<CLASS3>": <CLASS3_PROBABILITY>
}
}
},
{
"timestamp":<START_TS_OF_STEP>,
"results":{
<MODEL_NAME>:{
"result"::<MODEL_RESULT>,
"confidence":<MODEL_CONFIDENCE>
}
},
"raw": {
<MODEL_NAME>:{
"<CLASS1>": <CLASS1_PROBABILITY>,
"<CLASS2>": <CLASS2_PROBABILITY>,
"<CLASS3>": <CLASS3_PROBABILITY>
}
}
},
...
],
"summary":{...},
"transitions":{...}
}
}
}
An example with real values from the Arousal model:
{
"channels":{
"0":{
"time_series":[
{
"timestamp":0,
"results":{
"arousal":{
"result":"no_speech",
"confidence":0.748
}
},
"raw": {
"arousal": {
"high": 0.1143,
"low": 0.0034,
"neutral": 0.8823
}
}
},
{
"timestamp":4096,
"results":{
"arousal":{
"result":"no_speech",
"confidence":0.7105
}
},
"raw": {
"arousal": {
"high": 0.0078,
"low": 0.2143,
"neutral": 0.7779
}
}
},
{
"timestamp":8192,
"results":{
"arousal":{
"result":"high",
"confidence":0.9926
}
},
"raw": {
"arousal": {
"high": 0.9926,
"low": 0.0071,
"neutral": 0.0003
}
}
},
...
],
"summary":{...},
"transitions":{...}
}
}
}
summary
The summary key holds file-level calculations which summarise the prevalence of each class for a corresponding model.
{
"channels":{
<CHANNEL_IDENTIFIER>:{
"time_series":[],
"summary":{
<MODEL_NAME>:{
"<CLASS1>_fraction":0.5416,
"<CLASS2>_fraction":0.2609,
"<CLASS3>_fraction":0.1975
}
},
"transitions":{...}
}
}
}
An example with both the arousal and speech models:
{
"channels":{
"0":{
"time_series":[],
"summary":{
"speech":{
"music_fraction":0.5416,
"other_fraction":0.2609,
"speech_fraction":0.1975,
"silence_fraction": 0.0
},
"arousal":{
"high_fraction":0.1793,
"low_fraction":0.0071,
"neutral_fraction":0.0111,
"no_speech_fraction":0.8025,
"silence_fraction": 0.0
}
},
"transitions":{...}
}
}
}
transitions
The transitions key represents a summary of the timeseries. The idea is to indicate the starting time and ending time of each segment of audio, effectively collapsing time steps with the same model output. For more details on interpretation see the description of transitions in the corresponding model section.
Below is the output format, with an array of transitions for each model. Each element in the array represent an uninterrupted segment of the audio classified with the same value - e.g. music between from timestamp_start to timestamp_end.
{
"channels":{
<CHANNEL_IDENTIFIER>:{
"time_series":[],
"summary":{},
"transitions":{
<MODEL_NAME>:[
{
"timestamp_start":<START_TS_OF_SEGMENT>,
"timestamp_end":<END_TS_OF_SEGMENT>,
"result":<MODEL_RESULT_FOR_SEGMENT>,
"confidence":<MODEL_CONFIDENCE>
},
{
"timestamp_start":<START_TS_OF_SEGMENT>,
"timestamp_end":<END_TS_OF_SEGMENT>,
"result":<MODEL_RESULT_FOR_SEGMENT>,
"confidence":<MODEL_CONFIDENCE>
},
...
]
}
}
}
In the example with real data, you can see a music segment, followed by a speech segment.
{
"channels":{
"0":{
"time_series":[],
"summary":{},
"transitions":{
"speech":[
{
"timestamp_start":0,
"timestamp_end":6144,
"result":"music",
"confidence":0.7751
},
{
"timestamp_start":6144,
"timestamp_end":8128,
"result":"speech",
"confidence":0.5445
},
...
]
}
}
}
Complete example
Below you can find a complete example with speech and arousal models. You can paste it in https://jsonformatter.curiousconcept.com/ to explore it more easily.
The function call used to generate it is:
output = engine.process_file(
filename="vlog_2.wav",
models=[engine.models.Speech, engine.models.Arousal],
output_period=4096,
channel=0,
use_chunking=True,
include_summary=True,
include_transitions=True,
include_raw_values=True,
volume_threshold=0
)
Complete output:
{
"channels":{
"0":{
"time_series":[
{
"timestamp": 0,
"results":{
"speech": {
"result": "other",
"confidence": 0.6094
},
"arousal": {
"result": "no_speech",
"confidence": 0.6094
}
},
"raw": {
"speech": {
"music": 0.1735,
"other": 0.6094,
"speech": 0.2171
},
"arousal": {
"high": 0.2926,
"low": 0.1071,
"neutral": 0.6003
}
}
},
{
"timestamp": 4096,
"results":{
"speech": {
"result": "other",
"confidence": 0.6441
},
"arousal": {
"result": "no_speech",
"confidence": 0.6441
}
},
"raw": {
"speech": {
"music": 0.1357,
"other": 0.6441,
"speech": 0.2202
},
"arousal": {
"high": 0.3461,
"low": 0.201,
"neutral": 0.4529
}
}
},
{
"timestamp": 8192,
"results":{
"speech": {
"result": "speech",
"confidence": 0.5952
},
"arousal": {
"result": "high",
"confidence": 0.9992
}
},
"raw": {
"speech": {
"music": 0.1034,
"other": 0.3014,
"speech": 0.5952
},
"arousal": {
"high": 0.9992,
"low": 0.0002,
"neutral": 0.0006
}
}
}
],
"summary": {
"speech": {
"music_fraction": 0.0,
"other_fraction": 0.6543,
"speech_fraction": 0.3457,
"silence_fraction": 0.0
},
"arousal": {
"high_fraction": 0.3148,
"low_fraction": 0.0185,
"neutral_fraction": 0.0123,
"no_speech_fraction": 0.6543,
"silence_fraction": 0.0
}
},
"transitions": {
"speech": [
{
"timestamp_start": 0,
"timestamp_end": 64,
"result": "other",
"confidence": 0.779
},
{
"timestamp_start": 64,
"timestamp_end": 128,
"result": "speech",
"confidence": 0.6974
},
{
"timestamp_start": 128,
"timestamp_end": 448,
"result": "other",
"confidence": 0.5428
},
{
"timestamp_start": 448,
"timestamp_end": 512,
"result": "speech",
"confidence": 0.7051
},
{
"timestamp_start": 512,
"timestamp_end": 704,
"result": "other",
"confidence": 0.6106
},
{
"timestamp_start": 704,
"timestamp_end": 1152,
"result": "speech",
"confidence": 0.6448
},
{
"timestamp_start": 1152,
"timestamp_end": 1280,
"result": "other",
"confidence": 0.5515
},
{
"timestamp_start": 1280,
"timestamp_end": 1344,
"result": "speech",
"confidence": 0.5107
},
{
"timestamp_start": 1344,
"timestamp_end": 2816,
"result": "other",
"confidence": 0.7687
},
{
"timestamp_start": 2816,
"timestamp_end": 2880,
"result": "speech",
"confidence": 0.4946
},
{
"timestamp_start": 2880,
"timestamp_end": 2944,
"result": "other",
"confidence": 0.6252
},
{
"timestamp_start": 2944,
"timestamp_end": 3072,
"result": "speech",
"confidence": 0.6283
},
{
"timestamp_start": 3072,
"timestamp_end": 3200,
"result": "other",
"confidence": 0.572
},
{
"timestamp_start": 3200,
"timestamp_end": 3648,
"result": "speech",
"confidence": 0.6887
},
{
"timestamp_start": 3648,
"timestamp_end": 4864,
"result": "other",
"confidence": 0.8488
},
{
"timestamp_start": 4864,
"timestamp_end": 4992,
"result": "speech",
"confidence": 0.5812
},
{
"timestamp_start": 4992,
"timestamp_end": 5760,
"result": "other",
"confidence": 0.6926
},
{
"timestamp_start": 5760,
"timestamp_end": 5824,
"result": "speech",
"confidence": 0.5013
},
{
"timestamp_start": 5824,
"timestamp_end": 6144,
"result": "other",
"confidence": 0.5425
},
{
"timestamp_start": 6144,
"timestamp_end": 6272,
"result": "speech",
"confidence": 0.506
},
{
"timestamp_start": 6272,
"timestamp_end": 6464,
"result": "other",
"confidence": 0.5957
},
{
"timestamp_start": 6464,
"timestamp_end": 6656,
"result": "speech",
"confidence": 0.5601
},
{
"timestamp_start": 6656,
"timestamp_end": 7936,
"result": "other",
"confidence": 0.6909
},
{
"timestamp_start": 7936,
"timestamp_end": 8512,
"result": "speech",
"confidence": 0.6191
},
{
"timestamp_start": 8512,
"timestamp_end": 8576,
"result": "other",
"confidence": 0.5203
},
{
"timestamp_start": 8576,
"timestamp_end": 8768,
"result": "speech",
"confidence": 0.5293
},
{
"timestamp_start": 8768,
"timestamp_end": 8896,
"result": "other",
"confidence": 0.5398
},
{
"timestamp_start": 8896,
"timestamp_end": 8960,
"result": "speech",
"confidence": 0.4743
},
{
"timestamp_start": 8960,
"timestamp_end": 9408,
"result": "other",
"confidence": 0.6112
},
{
"timestamp_start": 9408,
"timestamp_end": 10368,
"result": "speech",
"confidence": 0.7707
}
],
"arousal": [
{
"timestamp_start": 0,
"timestamp_end": 64,
"result": "no_speech",
"confidence": 0.779
},
{
"timestamp_start": 64,
"timestamp_end": 128,
"result": "high",
"confidence": 0.492
},
{
"timestamp_start": 128,
"timestamp_end": 448,
"result": "no_speech",
"confidence": 0.5428
},
{
"timestamp_start": 448,
"timestamp_end": 512,
"result": "high",
"confidence": 0.8254
},
{
"timestamp_start": 512,
"timestamp_end": 704,
"result": "no_speech",
"confidence": 0.6106
},
{
"timestamp_start": 704,
"timestamp_end": 1152,
"result": "high",
"confidence": 0.9887
},
{
"timestamp_start": 1152,
"timestamp_end": 1280,
"result": "no_speech",
"confidence": 0.5515
},
{
"timestamp_start": 1280,
"timestamp_end": 1344,
"result": "high",
"confidence": 0.9965
},
{
"timestamp_start": 1344,
"timestamp_end": 2816,
"result": "no_speech",
"confidence": 0.7687
},
{
"timestamp_start": 2816,
"timestamp_end": 2880,
"result": "low",
"confidence": 0.8157
},
{
"timestamp_start": 2880,
"timestamp_end": 2944,
"result": "no_speech",
"confidence": 0.6252
},
{
"timestamp_start": 2944,
"timestamp_end": 3072,
"result": "low",
"confidence": 0.6866
},
{
"timestamp_start": 3072,
"timestamp_end": 3200,
"result": "no_speech",
"confidence": 0.572
},
{
"timestamp_start": 3200,
"timestamp_end": 3648,
"result": "high",
"confidence": 0.8684
},
{
"timestamp_start": 3648,
"timestamp_end": 4864,
"result": "no_speech",
"confidence": 0.8488
},
{
"timestamp_start": 4864,
"timestamp_end": 4992,
"result": "neutral",
"confidence": 0.7727
},
{
"timestamp_start": 4992,
"timestamp_end": 5760,
"result": "no_speech",
"confidence": 0.6926
},
{
"timestamp_start": 5760,
"timestamp_end": 5824,
"result": "high",
"confidence": 0.9167
},
{
"timestamp_start": 5824,
"timestamp_end": 6144,
"result": "no_speech",
"confidence": 0.5425
},
{
"timestamp_start": 6144,
"timestamp_end": 6272,
"result":" high",
"confidence": 0.9392
},
{
"timestamp_start": 6272,
"timestamp_end": 6464,
"result": "no_speech",
"confidence": 0.5957
},
{
"timestamp_start": 6464,
"timestamp_end": 6656,
"result": "high",
"confidence": 0.9931
},
{
"timestamp_start": 6656,
"timestamp_end": 7936,
"result": "no_speech",
"confidence": 0.6909
},
{
"timestamp_start": 7936,
"timestamp_end": 8512,
"result": "high",
"confidence": 0.9999
},
{
"timestamp_start": 8512,
"timestamp_end": 8576,
"result": "no_speech",
"confidence": 0.5203
},
{
"timestamp_start": 8576,
"timestamp_end": 8768,
"result":" high",
"confidence": 0.9992
},
{
"timestamp_start": 8768,
"timestamp_end": 8896,
"result": "no_speech",
"confidence": 0.5398
},
{
"timestamp_start": 8896,
"timestamp_end": 8960,
"result": "high",
"confidence": 1.0
},
{
"timestamp_start": 8960,
"timestamp_end": 9408,
"result": "no_speech",
"confidence": 0.6112
},
{
"timestamp_start": 9408,
"timestamp_end": 10368,
"result": "high",
"confidence": 0.999
}
]
}
}
}
}