Tooling for producing Italian model (public release available) for DeepSpeech and text corpus
After around a year since the last update a lot of changes have been made to the model and scripts Mainly there was a work to improve the audio+text dataset importer and bump to DeepSpeech 0.9.3.
It isn't a stable release as we don't have time now to do a proper release and also because there it will be soon the new CV dataset and now italian will have more than 300 hours compared to the version used to generate this.
For instructions how it was generated, parameters and other stuff check https://github.com/MozillaItalia/DeepSpeech-Italian-Model/wiki/Training-Notes-DeepSpeech-0.9.3-(2021.07.22-pre-release)
Total 475h
Available in 2 version transfer used transfer learning form the official English model release by mozilla and other one is from scratch .
This release was not possible without @eziolotta that did... everything! Me (@mte90) worked on the project management side about the model and with the help for the server offered by the Turin university we were able to do everything.
CC0 as public domain.
After 5 months we release a new model with a lot of improvements!
.env
files to try different parameters https://github.com/MozillaItalia/DeepSpeech-Italian-Model/tree/master/DeepSpeech/env_files
Available in 2 version transfer
used transfer learning form the official English model release by mozilla and other one is from scratch .
For transfer learning model:
Check the readme about the usage
This release wasn't possible without the huge work of @nefastosaturo on the docker and DS side other than generating the new model.
Me (@mte90) worked on the project management side about the model and @astrastefania with the help for the server offered by the Turin university we were able to do everything.
CC0 as public domain.
7 days ago we released the first version of Mitads and you can find all the information here.
The difference with previous version:
11922393
lines instead of 11828730
CC0 as public domain.
First official release of the Mitads text corpus!
Mitads is an Italian text corpus with sentences extracted from discussions, chats, books to get a kind of spoken Italian that can be used with AI like DeepSpeech.
This dataset is released as Public Domain
, it is generated with the scripts available at https://github.com/MozillaItalia/DeepSpeech-Italian-Model/tree/master/MITADS and is based on aggregating different datasets or resources that allow to be released in this aggregated way (basically it isn't possible to recreate from this the original datasets).
As it is a generated on-the-fly we cannot release the file cache or file generated during the process (for license issues) except the final corpus with a log file.
This corpus doesn't include repeated sentences, we implemented various sanitization but this tasks is never ending and require your help to improve the quality of the corpus itself.
Every script in the Mitads folder is for a specific resource that handle the download and parsing with generating txt files.
Usually every script has a caching workflow of external resources to speed up the development and generation itself, with specific rules to ignore lines, words and so on.
It is included a python library that is used for common tasks along the various scripts.
There is a final Bash script that execute all of them, do a final sanitization, remove duplciate sentences and generate the final corpus.
11828730
Close the last tickets and integrate this corpus with the script to generate a new model version. In our internal discussions use a text corpus more similar to Italian that is spoken between people the words recognition should improve a lot.
After the official release we will evaluate how to improve the performance, quality and maybe found new dataset suitable for this project.
Check with @mozitabot on Telegram and join the Mozilla Italia Developers group (we talk italian there).
The zip file include the tensorflow and tflite version.
Check the readme about the usage
Require DeepSpeech 0.6.0a9!
Italian Wikipedia Dump compressed