Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing
This update adds preprocessing and transformations for 6 Chinese NLP tasks, including Machine Reading Comprehension, Semantic Matching, Named Entity Recognition, and Sentiment Analysis.
It provides 15 universal transformations and 12 specific transformations.
Now support transformations of Neural Machine Translation transformation between English and German.
Now support transformations of Word Sense Disambiguation.
Now support transformations of the Winograd Schema Challenge.
Update requirements.
Update tutorial docs to synchronize with toolset version.
1> Optimize the installation , remove the textattack in the requirements. Because textattack relies on too many packages which may cause the failure of installation. It is recommended to install the package manually for adversarial attack.
2> Speed up the loading process of textflint from 1 minute to 3 seconds.
Input layer: receives textual datasets and models as input, represented as Dataset and FlintModel separately.
DataSet: a container for Sample, provides efficiently and handily operation interfaces for Sample. Dataset supports loading, verification, and saving data in Json or CSV format for various NLP tasks.FlintModel: a target model used in an adversarial attack.Generation layer: there are mainly four parts in generation layer:
Subpopulation: generates a subset of a DataSet.Transformation: transforms each sample of Dataset if it can be transformed.AttackRecipe: attacks the FlintModel and generate a DataSet of adversarial examples.Validator: verifies the quality of samples generated by Transformation and AttackRecipe.Report layer: analyzes model testing results and provides robustness report for users.