Processing the Switchboard Dialogue Act Corpus

Utilities for processing the Switchboard Dialogue Act Corpus for the purpose of dialogue act (DA) classification. The data is split into the original training and test sets suggested by the authors (1115 training and 19 test). The remaining 21 dialogues have been used as a validation set.

Scripts

The swda_to_text.py script processes all dialogues into a plain text format. Individual dialogues are saved into directories corresponding to the set they belong to (train, test, etc). All utterances in a particular set are also saved to a text file.

The utilities.py script contains various helper functions for loading/saving the data.

The process_transcript.py includes functions for processing each dialogue.

The swda_metadata.py generates various metadata from the processed dialogues and saves them as a dictionary to a pickle file. The words, labels and frequencies are also saved as plain text files in the /metadata directory.

Thanks to Christopher Potts for providing the raw data in .csv format and the swda.py script for processing the .csv data, both of which can be found here

Data Format

Utterance are tagged with the SWBD-DAMSL DA.

By default:

• Utterances are written one per line in the format Speaker | Utterance Text | Dialogue Act Tag.
• Setting the utterance_only_flag == True, will change the default output to only one utterance per line i.e. no speaker or DA tags.
• Utterances marked as Non-verbal ('x' tags) are removed i.e. 'Laughter' or 'Throat_clearing'.
• Utterances marked as Interrupted ('+' tags) and continued later are concatenated to make un-interrupted sentences.
• All disfluency annotations are removed i.e. '#', '<', '>', etc.

Example Utterances

A|What is the nature of your company's business?|qw

B|Well, it's actually, uh,|^h

B|we do oil well services.|sd

Dialogue Acts

Metadata

• Total number of utterances: 199740
• Max utterance length: 132
• Mean utterance length: 9.62
• Total Number of dialogues: 1155
• Max dialogue length: 457
• Mean dialogue length: 172.94
• Vocabulary size: 22302
• Number of labels: 41
• Number of speakers: 2

Train set

• Number of dialogues: 1115
• Max dialogue length: 457
• Mean dialogue length: 172.55
• Number of utterances: 192390

Test set

• Number of dialogues: 19
• Max dialogue length: 330
• Mean dialogue length: 214.63
• Number of utterances: 4078

Val set

• Number of dialogues: 21
• Max dialogue length: 299
• Mean dialogue length: 155.81
• Number of utterances: 3272

Keys and values for the metadata dictionary

• num_utterances = Total number of utterance in the full corpus.
• max_utterance_len = Number of words in the longest utterance in the corpus.
• mean_utterance_len = Average number of words in utterances.
• num_dialogues = Total number of dialogues in the corpus.
• max_dialogues_len = Number of utterances in the longest dialogue in the corpus.
• mean_dialogues_len = Average number of utterances in dialogues.
• word_freq = Dataframe with Word and Count columns.
• vocabulary = List of all words in vocabulary.
• vocabulary_size = Number of words in the vocabulary.
• label_freq = Dataframe containing all data in the Dialogue Acts table above.
• labels = List of all DA labels.
• num_labels = Number of labels used from the Switchboard data.
• speakers = List of all speakers.
• num_speakers = Number of speakers in the Switchboard data.

Each data set also has:

• <setname>_num_utterances = Number of utterances in the set.
• <setname>_num_dialogues = Number of dialogues in the set.
• <setname>_max_dialogue_len = Length of the longest dialogue in the set.
• <setname>_mean_dialogue_len = Mean length of dialogues in the set.