Emotion Postprocessing Outputs

This article describes the emotion aggregate postprocessing outputs produced by the Stim Admin interface while building a Dataset.

Explanation of Outputs

Data Structure

The output is provided as a .zip file containing 3 folders:

  • views (technical files in CSV and JSON format)
  • memo
  • charts

Memo is the name of the facial coding system that measures AUs. The bulk of the data is stored here. Charts contains some pre-rendered plots of key results. All key results are also available in interactive form in the Analysis part of Stim Admin.

File Formats

  • JSON (structured data for use in analysis by software)
  • CSV (data for use in MS Excel and other spreadsheet software)
  • PNG (images for visual confirmation and use in reports)

Key output files

The following files contain the most relevant data for further analysis.

File and location

File content

memo/timeseries_csv/

NNNNNN.csv

Individual emotion timeseries.

The emotion (AU) timeseries for each individual.  This data has been windowed and resampled, but not normalized. This is less immediately useful but handy if you want to drill down into the breakdown across the sample population.

memo/timeseries_aggregate/ absolute_smoothed.csv

Aggregate emotion timeseries, absolute values. This data has been windowed, resampled, binarized and aggregated. The data values represent the fraction and count of the population that are significantly expressing each emotion facial action unit (AU) at each time step.

memo/timeseries_aggregate/

relative.csv

Aggregate emotion timeseries, relative values. This is the same as the absolute, but relative to a moving baseline population fraction. This means it shows the changes in the frequency of expression, which is the key thing we want to maximize in most stimuli.

Key output charts

The following charts contain the most relevant data for further analysis. All charts are located in the charts folder.

File name

File content

peak_per_emotion.png

This shows the peak frequency of significant expression of each AU at any time. Higher is better. Note caveats on interpretation of AUs.

eyes_score_over_time.png

This shows the frequency of disengagement / lookaway events in the population over time. Higher = less interesting.

absolute_engagement_over_time.png

Engagement is the max frequency of significant emotional expression in the population. This plot is relative to a fixed baseline.

relative_engagement_over_time.png

As above, but relative to a moving baseline i.e. changes in expression frequency over time.

population_over_time.png

This plot is mainly a sanity check that the population sample is stable and large enough at all times. It is normal to lose a few participants during a session.

say_vs_do.png

This is a quadrant plot that combines effects and engagement.

wiggle_[relative | absolute].png

This is a stacked timeseries plot. Each colour shows one AU. The thickness of each colour is determined by the frequency of significant expression of that AU at each point in time during stim presentation. Frequencies here are relative to a fixed baseline.

zero_[relative | absolute].png

As above, but relative to a moving baseline frequency, meaning that significant changes in the frequency of AU expression are displayed.

e-score.png

A pie chart showing the breakdown of the 3 score contributions in the E-Score.

Interactive dataset results

All the outputs produced here, plus heatmaps, video and AOI statistics, will be used in our interactive results interface. Once you have created a Dataset, you can explore the results using the Analysis section of the Stim Admin interface.