Processing the Meeting Recorder Dialogue Act Corpus

Utilities for Processing the Meeting Recorder Dialogue Act Corpus outlined in this paper by Shriberg, E. et al.(2004) for the purpose of dialogue act (DA) classification. The data can also be downloaded here. The data is split into the original training and test sets suggested by the authors. There were two unused dialogues and these were added to the evaluation and test sets.

Scripts

The mrda_to_text.py script processes all dialogues into a plain text format. Individual dialogues are saved into directories corresponding to the set they belong to (train, test, etc). All utterances in a particular set are also saved to a text file.

The utilities.py script contains various helper functions for loading/saving the data.

The process_transcript.py includes functions for processing each dialogue.

The mrda_metadata.py generates various metadata from the processed dialogues and saves them as a dictionary to a pickle file. The words, labels and frequencies are also saved as plain text files in the /metadata directory.

Data Format

Utterance are tagged with the MRDA tagset, which is a variation of the SWBD-DAMSL DA. The original MRDA label construction allowed DA to be concatenated in the form <general tag> ^ <specific tag> . <disruptive from> (see the MRDA manual).

There are three sets of DA included:

• Basic, collapses all DA into 5 labels outlined in the original documentation DA maps (see the basic DA map file).
• General, uses the 12 <general tag> described in the MRDA manual.
• Full, uses only the first <specific tag> from the original labels, 52 in total.

By default:

• Utterances are written one per line in the format Speaker | Utterance Text | Basic DA Tag | General DA Tag | Full DA Tag.
• Setting the utterance_only_flag == True, will change the default output to only one utterance per line i.e. no speaker or DA tags.
• Utterances marked as Non-verbal ('x' tags) are removed i.e. 'Laughter' or 'Throat_clearing'.
• Utterances marked as Non-labeled ('z' tags) are removed.
• Interrupted, Abandoned and Uninterpretable tags are collapsed into one tag ('%').
• All disfluency annotations are removed i.e. '#', '<', '>', etc.

Example Utterances

fe016|okay.|F|fg|fg

fe016|so um|F|fh|fh

fe016|i was going to try to get out of here like in half an hour.|S|s|rt

Dialogue Acts

Basic Labels

General Labels

Full Labels

Metadata

• Total number of utterances: 108202
• Max utterance length: 85
• Mean utterance length: 8.01
• Total Number of dialogues: 75
• Max dialogue length: 3391
• Mean dialogue length: 1442.69
• Vocabulary size: 10866
• Number of basic labels: 5
• Number of general labels: 12
• Number of full labels: 52
• Number of speakers: 52

Train set

• Number of dialogues: 51
• Max dialogue length: 3391
• Mean dialogue length: 1471.9
• Number of utterances: 75067

Test set

• Number of dialogues: 12
• Max dialogue length: 2028
• Mean dialogue length: 1391.83
• Number of utterances: 16702

Val set

• Number of dialogues: 12
• Max dialogue length: 1969
• Mean dialogue length: 1369.42
• Number of utterances: 16433

Keys and values for the metadata dictionary

• num_utterances = Total number of utterance in the full corpus.
• max_utterance_len = Number of words in the longest utterance in the corpus.
• mean_utterance_len = Average number of words in utterances.
• num_dialogues = Total number of dialogues in the corpus.
• max_dialogues_len = Number of utterances in the longest dialogue in the corpus.
• mean_dialogues_len = Average number of utterances in dialogues.
• word_freq = Dataframe with Word and Count columns.
• vocabulary = List of all words in vocabulary.
• vocabulary_size = Number of words in the vocabulary.
• speakers = List of all speakers.
• num_speakers = Number of speakers in the MRDA data.

Each DA label set (basic, general or full) also has:

• <setname>_label_freq = Dataframe containing all data in the sets Dialogue Acts table above.
• <setname>_labels = List of all DA labels.
• num_<setname>_labels = Number of labels used from each of the label sets.

Each data set also has:

• <setname>_num_utterances = Number of utterances in the set.
• <setname>_num_dialogues = Number of dialogues in the set.
• <setname>_max_dialogue_len = Length of the longest dialogue in the set.
• <setname>_mean_dialogue_len = Mean length of dialogues in the set.