Workflow

  1. Choose whether to code child or mom emotion pass first
  2. Download associated coding spreadsheets and videos from Box
  3. Score each pass according to definitions in Emotion Codes.
  4. Run check typo scripts
  5. Delete videos from local computer (and empty the trash), upload complete and checked spreadsheets to Box

Child emotion pass

 Datavyu Emotion Codes for Child

There are 3 columns associated with child emotion codes: id for child emotion, child emotion column, and a comments column for the child emotion coding.

Child emotion id Column

Column name: emo_id_child

Code: (<lab_id>, <childemo_coder>, <childemo_date>, <childemo_mins>,  <relchildemo_coder>, <relchildemo_date>)

Code the emo_id_child column according to the definitions for coding the id column.

Only code within the onset/offset of the ID column, which is the selected hour for that visit. If the first frame includes a coding event then start the first cell at that onset time (do not go backward in the video to find the “true” onset). Similarly, if the last event continues beyond the offset of ID, end the cell at the offset of ID.

Child emotion Column

Column name: childemo

Code: (<emotion_pn>)

General Orientation

This code captures the times when the child is clearly displaying positive or negative emotion through facial expressions. Times when the child is in neutral emotion are not marked. Bouts of emotion are scored as events, where the grey spaces between emotion events mean the child is neutral or emotion is unclear. Coders also mark times as “missing” when the child’s head is not visible on camera and emotion could be coded. That is, we do not code missing based on the face (it’s too arduous and subjective what it means for the face to be missing); code missing when the head is off camera. Emotion shouldnever be coded based solely on vocal affect, unless the child is displaying clear and robust distress. That is, when the child’s face is not clearly visible, do not code positive emotion. We have found that positive affect is not reliably codeable from the voice alone and only clear, robust negative affect, such as crying, wailing, and screaming (not whimpering or frustration), is codeable from the voice alone.

Coders are watching/tagging the duration of each positive or negative emotion event by marking onset/offset times. To determine emotion, coders are watching the child’s face, not vocal affect alone. However, coders should keep audio on while coding in order to help differentiate talking from bouts of facial affect. To determine if emotion is not codeable/missing, coders are watching for when the head fully moves out of the camera view.

After completing the coding pass for the entire video, coders should rewatch the first 5 minutes in order to recheck their codes and ensure that they did not miss any positive or negative emotion bouts. Coders must do this because each child expresses positive and negative emotions differently, which coders become more attuned to after extensive experience coding for that child. Therefore, coders may have inaccurately coded earlier parts of the video.

Value List

‘p’ = positive emotion

‘n’ = negative emotion

‘.’ = child’s head is not visible on the video

Operational Definitions

Positive (p)

Code ‘p’ when the child is clearly displaying positive emotion (e.g., smiling). Code based off of the face and not off of the voice. Look for raising of the corners of the mouth, grinning and showing the teeth, along with closing of the eyes because of the raised cheeks. If there is any doubt that the child is showing positive emotion, then do not begin the code.

Positive emotion cannot be coded based on the voice alone, therefore positive emotion should never be scored when the child’s face is not visible. If there is a mismatch between facial expression and vocal affect (eg. child’s face is neutral but sounds like they are laughing), always defer to the facial expression (eg. do not code this as positive emotion bout).

Negative (n)

Code ‘n’ when the child is clearly displaying negative emotion (e.g., frowning, wincing). Code based off of the face. Look for lowering of the corners of the mouth, stretching and tautness of the lips, along with closing of the eyes because of the furrowed brow. If there is any doubt that the child is showing negative emotion, then do not begin the code.

If the child’s face is not visible, but they are audibly displaying clear, high valence negative affect (e.g., screaming, crying, wailing), negative emotion can be coded based on the voice alone. Unclear or lower valence negative affect (e.g., whimpering, moaning, sulking, frustration) should not be coded based on the voice alone. If there is a mismatch between facial expression and vocal affect (eg. child’s face is neutral but sounds like they are whining), always defer to the facial expression (eg. do not code this as negative emotion bout).

Setting the Onset & Offset of emotion:

Onset & Offset for ‘p’ or ‘n’

Onset of an emotion event is the first frame when the child is clearly displaying positive/negative emotion through the face. The coder is looking to identify when any lay person would absolutely agree the child is showing positive/negative emotion based on the face.

There is no criterion for how long an emotion event should be. It is easy for the coder to mark the first frame when they see clear positive/negative emotion, even if it ends a few frames later. Events that are later deemed “too brief” could be removed via scripting.

For cases when emotion code begins directly after missing (i.e., the head has not been visible and the first frame you can see the face the infant is clearly displaying positive/negative emotion): Code the offset of missing and onset of the emotion code as the first frame when the face reappears. Use the “0” key to set the onset of the emotion event and simultaneously set the offset of ‘missing’. We want to preserve the 1ms difference between ‘missing’ and the emotion code so we can know that that ‘missing’ event ended because of the onset of an emotion code.

Offset of a positive/negative emotion is the first frame the child is clearly back to a neutral emotion through the face. The coder is looking to identify when any lay person would absolutely agree the child is no longer showing any positive/negative emotion based on the face.

If the child’s face returns to neutral or is missing for less than 5 frames (150 ms) during one emotion code (e.g. positive, then neutral for 4 frames, then back to positive), continue the ‘p’ or ‘n’ code. The coder would have to expend unneeded effort to identify and tag those offsets and onsets time.

For cases when emotion code is ended by missing (i.e., the emotion event may or may not have ended, but the coder can no longer see any part of the child’s head): Code the offset as the first frame the head completely is visible (see ‘missing’ code below). Use the “0” key to set the offset of emotion and simultaneously code the onset of ‘missing’. We want to preserve the 1ms difference between the emotion code and ‘missing’ so we can know that that ‘missing’ event caused the offset of that emotion event.

Missing (.)

Start the code  ‘.’ for missing when child’s entire head is out of view (head could be occluded by furniture or mom, head not on camera view, or whole child missing from view). The coder should not start code missing if they are able to see any part of the face or head. There is no minimum duration for coding missing bouts.

Stop the code ‘.’ when any part of the child’s head (forehead, cheek, chin) comes back in view. Even if part of the head comes into view (hair, neck, ears).

(Note 1: While it is not possible to code emotion when none of the head (from the neck up) is visible, it is too labor intensive for coders to code every time the face is in or out of view. In order to capture when emotion is not codeable in the most streamlined way, coders should only start to code missing when the whole head is not visible. The goal is to minimize coder effort, but mark out many periods of time when it would not be possible to code facial emotion.

Note 2: If they are wearing the clothing then do not code as missing. If it's clothing off their body (external piece of clothing) then code as missing.)

Setting the Onset & Offset of missing:

Onset is the first frame in which the coder clearly cannot see any part of the head.

Offset is the first frame in which the coder can see any part of the head again.

If the child’s head was ‘missing’ and then reappeared for less than 5 frames, don’t stop the ‘.’ code to mark those few frames.

How to Code the Child Emotion Column

Play with #8-PLAY in real time (1x speed) until the child changes to clear positive or clear negative emotion or the head is not completely visible. Focus on the child’s face.Do not be distracted by what the child is saying or doing. Audio gives context and helps to distinguish talking from emoting.

Pause with #5-STOP once a clear change in emotion is identified or  when the child’s head is no longer visible at all. Shuttle back with #4-SHUTTLEBACK at 1/8-1/4x speed to identify the onset. Use the mouth and eyes as the guide to set the onset. Press ENTER to set the onset as the frame where any lay person would say that child is happy or sad. The coder may even feel happy or sad watching the child’s face; use this as a guide for onset.

Then hit #8-PLAY then #4-SHUTTLEBACK once to watch at 1/2x and look for the offset of that emotion or when the head comes completely back into view. For missing, if it seems like there may be a long stretch of missing (e.g. child has completely wandered out of the room) then watch at 1x or 2x speed. Pause when you identify the offset.

Hit #1-JOGBACK and #3-JOGFORWARD to tag the frame the child’s face is clearly not positive or not negative (returned to neutral) or the head is visible again to code emotion from.

Then return to real time (1x speed) with #8-PLAY to watch for the next event.

Check the final coded pass

When finished coding, run the script to check for typos in the child emotion column. (Download the script from the “Scripts for Emotion Coding” folder on Box.) In the Datavyu menu select Script> Run Script. The script will flag any occurrence of typos that arise from an error in setting the onset or offset (not coded, cell with no duration, overlapping cells), consecutive ‘p’ cells or consecutive ‘n’ that are less than 5 frames apart (and should be merged), and any invalid codes (any characters besides ‘p’, ‘n’, or ‘.’). Fix errors and run the script again. When all the errors have been resolved the script will inform that no errors are found, including errors in the ID column.

Mom emotion pass

Datavyu emotion Codes for Mom

There are 3 columns associated with mom emotion codes: id for mom emotion, mom emotion column, and a comments column for the mom emotion coding.

Mom emotion id Column

Column name: emo_id_mom

Code: (<lab_id>, <momemo_coder>, <momemo_date>, <momemo_mins>,  <relmomemo_coder>, <relmomemo_date>)

Code the emo_id_mom column according to the definitions for coding the id column.

Only code within the onset/offset of the ID column, which is the selected hour for that visit. If the first frame includes a coding event then start the first cell at that onset time (do not go backward in the video to find the “true” onset). Similarly, if the last event continues beyond the offset of ID, end the cell at the offset of ID.

Mom emotion Column

Column name: momemo

Code: (<emotion_pn>)

General Orientation

This code captures the times when the mom is clearly displaying positive or negative emotion through facial expressions. Times when the mom is in neutral emotion are not marked. Bouts of emotion are scored as events, where the grey spaces between emotion events mean the mom is neutral or emotion is unclear. Coders also mark times as “missing” when the mom’s head is not visible on camera and emotion could not be coded. That is, we do not code missing based on the face (it’s too arduous and subjective what it means for the face to be missing); code missing when the head is off camera. Emotion may never be coded based solely on vocal affect. That is, when the mom’s face is clearly not clearly visible, do not code positive or negative emotion.

Coders are watching/tagging the duration of each positive or negative emotion event by marking onset/offset times. To determine emotion, coders are watching the mom’s face, not vocal affect. However, coders should keep audio on while coding in order to help differentiate talking from bouts of facial affect. To determine if emotion is not codeable/missing, coders are watching for when the head fully moves out of the camera view.

After completing the coding pass for the entire video, coders should rewatch the first 5 minutes in order to recheck their codes and ensure that they did not miss any positive or negative emotion bouts. Coders must do this because each mom expresses positive and negative emotions differently, which coders become more attuned to after extensive experience coding for that mom. Therefore, coders may have inaccurately coded earlier parts of the video.

Value List

‘p’ = positive emotion

‘n’ = negative emotion

‘.’ = mom’s head is completely not visible

Operational Definitions

Positive (p)

Code ‘p’ when the mom is clearly displaying positive emotion (e.g., smiling). Code based off of the face and not off of the voice. Look for raising of the corners of the mouth, grinning and showing the teeth, along with closing of the eyes because of the raised cheeks. If there is any doubt that the mom is showing positive emotion, then do not begin the code.

Positive emotion cannot be coded based on the voice alone, therefore positive emotion should not be scored when the mom’s face is absolutely not visible. If there is a mismatch between facial expression and vocal affect (eg. mom’s face is neutral but sounds like they are laughing), always defer to the facial expression (eg. do not code this as positive emotion bout).

Mothers often will display “performative smiling” where they are acting happy, coaxing the child to smile by smiling, or other facial actions where they are making themselves smile. These would be coded as ‘p’ so long as they are clear and obvious; there is not a reliable or meaningful way to distinguish these from “genuine” smiles.

Negative (n)

Code ‘n’ when the mom is clearly displaying negative emotion (e.g., frowning, wincing). Code based off of the face and not off of the voice (unless in cases where the face is not visible). Look for lowering of the corners of the mouth, stretching and tautness of the lips, along with closing of the eyes because of the furrowed brow. If there is any doubt that the mom is showing negative emotion, then do not begin the code.

Negative emotion cannot be coded based on the voice alone, therefore. So negative emotion should never be scored when the mom’s face is not visible. If there is a mismatch between facial expression and vocal affect (eg. mom’s face is neutral but sounds like they are whining), always defer to the facial expression (eg. do not code this as negative emotion bout).

Mothers sometimes will display “performative” frowning or pouting where they are mimicking the child’s negative affect, empathizing with the child, or pretending to be a character in a book they are reading, or putting on a frown. Code these as ‘n’ as it’s not reliable or meaningful to distinguish these from “genuine” negative affect.

 

Setting the Onset & Offset of emotion:

Onset of an emotion event is the first frame when the mom is clearly displaying positive/negative emotion through the face. When coding ‘n’ from voice during missing time, set the onset when the negative voice starts and end the ‘n’ code when the voice ends. The coder is looking to identify when any lay person would absolutely agree the mom is showing positive/negative emotion based on the face.

There is no criterion for how long an emotion event should be. It is easy for the coder to mark the first frame when they see clear positive/negative emotion, even if it ends a few frames later. Events that are later deemed “too brief” could be removed via scripting.

For cases when emotion code begins directly after missing (i.e., the head has not been visible and the first frame you can see the face the infant is clearly displaying positive/negative emotion): Code the offset of missing and onset of the emotion code as the first frame when the face reappears. Use the “0” key to set the onset of the emotion event and simultaneously set the offset of ‘missing’. We want to preserve the 1ms difference between ‘missing’ and the emotion code so we can know that that ‘missing’ event was ended because of the onset of an emotion code.

Offset of a positive/negative emotion is the first frame the mom is clearly back to a neutral emotion through the face. The coder is looking to identify when any lay person would absolutely agree the mom is no longer showing any positive/negative emotion based on the face.

If the mom’s face returns to neutral or is missing for less than 5 frames during one emotion code (e.g. positive, then neutral for 4 frames, then back to positive), continue the ‘p’ or ‘n’ code. The coder would have to expend unneeded effort to identify and tag those offsets and onsets time, since reliability does not need to be frame accurate.

For cases when emotion code is ended by missing (i.e., the emotion event may or may not have ended, but the coder can no longer see any part of the mom’s head): Code the offset as the first frame the head completely is visible (see ‘missing’ code below). Use “0” key to set the offset of emotion and simultaneously code the onset of ‘missing’. We want to preserve the 1ms difference between the emotion code and ‘missing’ so we can know that that ‘missing’ event caused the offset of that emotion event.

Missing (.)

Code ‘.’ for missing when the mom’s entire head is out of view (head could be occluded by furniture or child, head not on camera view, or whole mom missing from view). It’s too labor intensive to code every time the face is in or out of view, but we do know that when none of the head is not visible (from the neck up) coding emotion is impossible. So to capture when emotion is not codeable in the most streamlined way, we only code missing when the whole head is not visible. If the coder could see any part of the face or head, then do not code missing.

Setting the Onset & Offset of Missing:

Onset is the first frame in which the coder clearly cannot see any part of the head.

Offset is the first frame in which the coder can see the head again.

If the mom’s head was ‘missing’ and then reappeared for less than 5 frames, don’t stop the ‘.’ code to mark those few frames.

How to Code the Mom Emotion Column

Play with #8-PLAY in real time (1x speed) until the mom changes to clear positive or clear negative emotion or the face is not completely visible. Focus on the mom’s face.Do not be distracted by what the mom is saying or doing. Audio gives context and helps to distinguish talking from emoting.

Pause with #5-STOP once a clear change in emotion is identified or when the mom’s head is no longer visible at all. Shuttle back with #4-SHUTTLEBACK at 1/8-1/4x speed to identify the onset. Use the mouth and eyes as the guide to set the onset. Press ENTER to set the onset as the frame where any lay person would say that mom is happy or sad. The coder may even feel happy or sad watching the mom’s face; use this as a guide for onset.

Then hit #8-PLAY then #4-SHUTTLEBACK once to watch at 1/2x and look for the offset of that emotion or when the head comes completely back into view. For missing, if it seems like there may be a long stretch of missing (e.g. mom is on a different side of the room from the child) then watch at 1x or 2x speed. Pause when you identify the offset.

Hit #1-JOGBACK and #3-JOGFORWARD to tag the frame the mom’s face is clearly not positive or not negative (returned to neutral) or the head is visible again to code emotion from.

Then return to real time (1x speed) with #8-PLAY to watch for the next event.

Check the final coded pass

When finished coding, run the script to check for typos in the mom emotion column. The script will flag any occurrence of typos that arise from an error in setting the onset or offset (not coded, cell with no duration, overlapping cells), consecutive ‘p’ cells or consecutive ‘n’ that are less than 5 frames apart (and should be merged), and any invalid codes (any characters besides ‘p’, ‘n’, or ‘.’). Fix errors and run the script again. When all the errors have been resolved the script will inform that no errors are found.