Mindee Doctr Versions Save

docTR (Document Text Recognition) - a seamless, high-performing & accessible library for OCR-related tasks powered by Deep Learning.

v0.8.1

2 months ago

Note: doctr 0.8.1 requires either TensorFlow >= 2.11.0 or PyTorch >= 1.12.0.

What's Changed

v0.8.0

2 months ago

Note: doctr 0.8.0 requires either TensorFlow >= 2.11.0 or PyTorch >= 1.12.0.

What's Changed

Breaking Changes 🛠

  • db_resnet50_rotation (PyTorch) and linknet_resnet18_rotation (TensorFlow) are removed (All models can handle rotated documents now)
  • .show(doc) changed to .show()

New features

  • All models have pretrained checkpoints now by @odulcy-mindee
  • All detection models was retrained on rotated samples by @odulcy-mindee
  • Improved orientation detection for documents rotated between -90 and 90 degrees by @felixdittrich92
  • Conda deployment job & receipt added by @frgfm
  • Official docTR docker images are added by @odulcy-mindee => docker-images
  • New benchmarks and documentation improvements by @felixdittrich92
  • WildReceipt dataset added by @HamzaGbada
  • EarlyStopping callback added to all training scripts by @SkaarFacee
  • Hook mechanism added to ocr_predictor to maniplulate the detection predictions in the middle of the pipeline to your needs by @felixdittrich92

from doctr.model import ocr_predictor

class CustomHook:
    def __call__(self, loc_preds):
        # Manipulate the location predictions here
        # 1. The outpout structure needs to be the same as the input location predictions
        # 2. Be aware that the coordinates are relative and needs to be between 0 and 1
        return loc_preds

my_hook = CustomHook()

predictor = ocr_predictor(pretrained=True)
# Add a hook in the middle of the pipeline
predictor.add_hook(my_hook)
# You can also add multiple hooks which will be executed sequentially
for hook in [my_hook, my_hook, my_hook]:
    predictor.add_hook(hook)

What's Changed

Breaking Changes 🛠

New Features

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: https://github.com/mindee/doctr/compare/v0.7.0...v0.8.0

v0.7.0

8 months ago

Note: doctr 0.7.0 requires either TensorFlow >= 2.11.0 or PyTorch >= 1.12.0. Note: We will release the missing PyTorch checkpoints with 0.7.1

What's Changed

Breaking Changes 🛠

  • We changed the preserve_aspect_ratio parameter to True by default in https://github.com/mindee/doctr/pull/1279 => To restore the old behaviour you can pass preserve_aspect_ratio=False to the predictor instance

New features

  • Feat: Make detection training and inference Multiclass by @aminemindee in https://github.com/mindee/doctr/pull/1097
  • Now all TensorFlow models have pretrained weights by @odulcy-mindee
  • The docs was updated and model corresponding benchmarks was added by @felixdittrich92
  • Two new recognition models was added (ViTSTR and PARSeq) in both frameworks by @felixdittrich92 @nikokks

Add of the KIE predictor

The KIE predictor is a more flexible predictor compared to OCR as your detection model can detect multiple classes in a document. For example, you can have a detection model to detect just dates and adresses in a document.

The KIE predictor makes it possible to use detector with multiple classes with a recognition model and to have the whole pipeline already setup for you.

from doctr.io import DocumentFile
from doctr.models import kie_predictor

# Model
model = kie_predictor(det_arch='db_resnet50', reco_arch='crnn_vgg16_bn', pretrained=True)
# PDF
doc = DocumentFile.from_pdf("path/to/your/doc.pdf")
# Analyze
result = model(doc)

predictions = result.pages[0].predictions
for class_name in predictions.keys():
    list_predictions = predictions[class_name]
    for prediction in list_predictions:
        print(f"Prediction for {class_name}: {prediction}")

The KIE predictor results per page are in a dictionary format with each key representing a class name and it's value are the predictions for that class.

What's Changed

Breaking Changes 🛠

New Features

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: https://github.com/mindee/doctr/compare/v0.6.0...v0.7.0

v0.6.0

1 year ago

Highlights of the release:

Note: doctr 0.6.0 requires either TensorFlow >= 2.9.0 or PyTorch >= 1.8.0.

Full integration with Huggingface Hub (docTR meets Huggingface)

hf

  • Loading from hub:
from doctr.io import DocumentFile
from doctr.models import ocr_predictor, from_hub
image = DocumentFile.from_images(['data/example.jpg'])
# Load a custom detection model from huggingface hub
det_model = from_hub('Felix92/doctr-torch-db-mobilenet-v3-large')
# Load a custom recognition model from huggingface hub
reco_model = from_hub('Felix92/doctr-torch-crnn-mobilenet-v3-large-french')
# You can easily plug in this models to the OCR predictor
predictor = ocr_predictor(det_arch=det_model, reco_arch=reco_model)
result = predictor(image)
  • Pushing to the hub:
from doctr.models import recognition, login_to_hub, push_to_hf_hub
login_to_hub()
my_awesome_model = recognition.crnn_mobilenet_v3_large(pretrained=True)
push_to_hf_hub(my_awesome_model, model_name='doctr-crnn-mobilenet-v3-large-french-v1', task='recognition', arch='crnn_mobilenet_v3_large')

Documentation: https://mindee.github.io/doctr/using_doctr/sharing_models.html

Predefined datasets can be used also for recognition task

from doctr.datasets import CORD
# Crop boxes as is (can contain irregular)
train_set = CORD(train=True, download=True, recognition_task=True)
# Crop rotated boxes (always regular)
train_set = CORD(train=True, download=True, use_polygons=True, recognition_task=True)
img, target = train_set[0]

Documentation: https://mindee.github.io/doctr/using_doctr/using_datasets.html

New models (both frameworks)

  • classification: VisionTransformer (ViT)
  • recognition: Vision Transformer for Scene Text Recognition (ViTSTR)

Bug fixes recognition models

  • MASTER and SAR architectures are now operational in both frameworks (TensorFlow and PyTorch)

ONNX support (experimential)

  • All models can now be exported into ONNX format (only TF mobilenet left for 0.7.0)

NOTE: full production pipeline with ONNX / build is planned for 0.7.0 (the models can be only exported up to the logits without any post processing included)

Further features

  • our demo is now also PyTorch compatible, thanks to @odulcy-mindee
  • it is now possible to detect the language of the extracted text, thanks to @aminemindee

What's Changed

Breaking Changes 🛠

New Features

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: https://github.com/mindee/doctr/compare/v0.5.1...v0.6.0

v0.5.1

2 years ago

This minor release includes: improvement of the documentation thanks to @felixdittrich92, bugs fixed, support of rotation extended to Tensorflow backend, a switch from PyMuPDF to pypdfmium2 and a nice integration to the Hugginface Hub thanks to @fg-mindee !

Note: doctr 0.5.0 requires either TensorFlow 2.4.0 or PyTorch 1.8.0.

Highlights

Improvement of the documentation

The documentation has been improved adding a new theme, illustrations, and docstring has been completed and developed. This how it renders:

doc Capture d’écran de 2022-03-22 11-08-31

Rotated text detection extended to Tensorflow backend

We provide weights for the linknet_resnet18_rotation model which has been deeply modified: We implemented a new loss (based on Dice Loss and Focal Loss), we changed the computation of the targets so that polygons are shrunken the same way they are in the DBNet which improves highly the precision of the segmenter and we trained the model preserving the aspect ratio of the images. All these improvements led to much better results, and the pretrained model is now very robust.

Preserving the aspect ratio in the detection task

You can now choose to preserve the aspect ratio in the detection_predictor:

>>> from doctr.models import detection_predictor
>>> predictor = detection_predictor('db_resnet50_rotation', pretrained=True, assume_straight_pages=False, preserve_aspect_ratio=True)

This option can also be activated in the high level end-to-end predictor:

>>> from doctr.model import ocr_predictor
>>> model = ocr_predictor('linknet_resnet18_rotation', pretrained=True, assume_straight_pages=False, preserve_aspect_ratio=True)

Integration within the HugginFace Hub

The artefact detection model is now available on the HugginFace Hub, this is amazing:

Capture d’écran de 2022-03-22 11-33-14

On DocTR, you can now use the .from_hub() method so that those 2 snippets are equivalent:

# Pretrained
from doctr.models.obj_detection import fasterrcnn_mobilenet_v3_large_fpn
model = fasterrcnn_mobilenet_v3_large_fpn(pretrained=True)

and:

# HF Hub
from doctr.models.obj_detection.factory import from_hub
model = from_hub("mindee/fasterrcnn_mobilenet_v3_large_fpn")

Breaking changes

Replacing the PyMuPDF dependency with pypdfmium2 which is license compatible

We replaced for the PyMuPDF dependency with pypdfmium2 for a license-compatibility issue, so we loose the word and objects extraction from source pdf which was done with PyMuPDF. It wasn't used in any models so it is not a big issue, but anyway we will work in the future to re-integrate such a feature.

Full changelog

What's Changed

Breaking Changes 🛠

New Features

Bug Fixes

Improvements

Miscellaneous

New Contributors

Full Changelog: https://github.com/mindee/doctr/compare/v0.5.0...v0.5.1

v0.5.0

2 years ago

This release adds support of rotated documents, and extends both the model & dataset zoos.

Note: doctr 0.5.0 requires either TensorFlow 2.4.0 or PyTorch 1.8.0.

Highlights

:upside_down_face: :smiley: Rotation-aware text detection :upside_down_face: :smiley:

It's no secret: this release focus was to bring the same level of performance to rotated documents!

predictions

docTR is meant to be your best tool for seamless document processing, and it couldn't do without supporting a very natural & common augmentation of input documents. This large project was subdivided into three parts:

Straightening pages before text detection

Developing a heuristic-based method to estimate the page skew, and rotate it before forwarding it to any deep learning model. Our thanks to @Rob192 for his contribution on this part :pray:

This behaviour can be enabled to avoid retraining the text detection models. However, the heuristics approach has its limits in terms of robustness.

Text detection training with rotated images

doctr_sample

The core of this project was to enable our text detection models to produce non-degraded heatmaps & localization candidates when processing a rotated page.

Crop orientation resolution

rot2

Finally, once the localization candidates have been extracted, there is no saying that this localization candidate will read from left to right. In order to remove this doubt, a lightweight image orientation classifier was added to refine the crops that will be sent to text recognition!

:zebra: A wider pretrained classification model zoo :zebra:

The stability of trainings in deep learning for complex tasks has mostly been helped by leveraging transfer learning. As such, OCR tasks usually require a backbone as a feature extractor. For this reason, all checkpoints of classification models in both PyTorch & TensorFlow have been updated :rocket: Those were trained using our synthetic character classification dataset, for more details cf. Character classification training

:framed_picture: New public datasets join the fray

Thanks to @felixdittrich92, the list of supported datasets has considerably grown :partying_face: This includes widely popular datasets used for benchmarks on OCR-related tasks, you can find the full list over here :point_right: #587

Synthetic text recognition dataset

Additionally, we followed up on the existing CharGenerator by introducing WordGenerator:

  • generates an image of word of length randomly sampled within a specified range, with characters randomly sampled from the specified vocab.
  • you can even pass a list of fonts so that each word font family is randomly picked among them

Below are some samples using a font_size=32: wordgenerator_sample

:bookmark_tabs: New notebooks

Two new notebooks have made their way into the documentation:

  • producing searchable PDFs from docTR analysis results
  • introduction to document artefact detection (QR code, bar codes, ID pictures, etc.) with docTR

image

Breaking changes

Revamp of classification models

With the retraining of all classification backbones, several changes have been introduced:

  • Model naming: linknet16 --> linknet_resnet18
  • Architecture changes: all classification backbones are available with a classification head now.

Enforcing relative coordinates in datasets

In order to unify our data pipelines, we forced the conversion to relative coordinates on all datasets!

0.4.1 0.5.0
>>> from doctr.datasets import FUNSD
>>> ds = FUNSD(train=True, download=True)
>>> img, target = ds[0]
>>> print(target['boxes'].dtype, target['boxes'].max())
(dtype('int64'), 862)		 >>> from doctr.datasets import FUNSD
>>> ds = FUNSD(train=True, download=True)
>>> img, target = ds[0]
>>> print(target['boxes'].dtype, target['boxes'].max())
(dtype('float32'), 0.98341835)

Full changelog

Breaking Changes 🛠

New Features

Bug Fixes

Improvements

Miscellaneous

Full Changelog: https://github.com/mindee/doctr/compare/v0.4.1...v0.5.0

v0.4.1

2 years ago

This patch release brings the support of AMP for PyTorch training to docTR along with artefact object detection.

Note: doctr 0.4.1 requires either TensorFlow 2.4.0 or PyTorch 1.8.0.

Highlights

Automatic Mixed Precision (AMP) :zap:

Training scripts with PyTorch back-end now benefit from AMP to reduce the RAM footprint and potentially increase the maximum batch size! This comes especially handy on text detection which require high spatial resolution inputs!

Artefact detection :flying_saucer:

Document understanding goes beyond textual elements, as information can be encoded in other visual forms. For this reason, we have extended the range of supported tasks by adding object detection. This will be focused on non-textual elements in documents, including QR codes, barcodes, ID pictures, and logos.

Here are some early results:

2x3_art(1)

This release comes with a training & validation set DocArtefacts, and a reference training script. Keep an eye for models we will be releasing in the next release!

Get more of docTR with Colab tutorials :book:

You've been waiting for it, from now on, we will be adding regularly new tutorials for docTR in the form of jupyter notebooks that you can open and run locally or on Google Colab for instance!

Check the new page in the documentation to have an updated list of all our community notebooks: https://mindee.github.io/doctr/latest/notebooks.html

Breaking changes

Deprecated support of FP16 for datasets

Float-precision can be leveraged in deep learning to decrease the RAM footprint of trainings. The common data type float32 has a lower resolution counterpart float16 which is usually only supported on GPU for common deep learning operations. Initially, we were planning to make all our operations available in both to reduce memory footprint in the end.

However, with the latest development of Deep Learning frameworks, and their Automatic Mixed Precision mechanism, this isn't required anymore and only adds more constraints on the development side. We thus deprecated this feature from our datasets and predictors:

0.4.0 0.4.1
>>> from doctr.datasets import FUNSD
>>> ds = FUNSD(train=True, download=True, fp16=True)
>>> print(getattr(ds, "fp16"))
True		 >>> from doctr.datasets import FUNSD
>>> ds = FUNSD(train=True, download=True)
>>> print(getattr(ds, "fp16"))
None

Detailed changes

New features

  • Adds Arabic to supported vocabs in #514 (@mzeidhassan)
  • Adds XML export method to DocumentBuilder in #544 (@felixdittrich92)
  • Adds flags to control the behaviour with rotated elements in #551 (@charlesmindee)
  • Adds unittest to ensure headers are correct in #556 (@fg-mindee)
  • Adds isort ordering & dedicated CI check in #557 (@fg-mindee)
  • Adds IIIT-5K to supported datasets in #589 (@felixdittrich92)
  • Adds support of AMP to all PyTorch training scripts in #604 (@fg-mindee)
  • Adds DocArtefacts dataset for object detection on non-textual elements in #583 (@SiddhantBahuguna)
  • Speeds up CTC decoding in PyTorch by x10 in #633 (@fg-mindee)
  • Added train script for artefact detection in #593 (@SiddhantBahuguna)
  • Added GPU support for classification and improve memory pinning in #629 (@fg-mindee)
  • Added an object detection metric in #628 (@fg-mindee)
  • Split DocArtefacts into subsets and updated its class mapping in #601 (@fg-mindee)
  • Added README specific for the API with route examples in #612 (@fg-mindee)
  • Added SVT dataset integration in #620 (@felixdittrich92)
  • Added links to tutorial notebooks in the documentation in #619 (@fg-mindee)
  • Added new architectures to model selection in demo in #600 (@fg-mindee)
  • Add det/reco_predictor arch in OCRPredictor.__repr__ in #595 (@RBMindee)
  • Improves coverage by adding missing unittests in #545 (@fg-mindee)
  • Resolve both lines and blocks by default when building a doc in #548 (@charlesmindee)
  • Relocated test/ to tests/ and made contribution process easier in #598 (@fg-mindee)
  • Fixed Makefile by converting spaces to tabs in #615 (@fg-mindee)
  • Updated flake8 config to spot unused imports & undefined variables in #623 (@fg-mindee)
  • Adds 2 new rotation flags in the ocr_predictor in #632 (@charlesmindee)

Bug fixes

  • Fixed evaluation script clipping issue in #522 (@charlesmindee)
  • Fixed API template issues with new httpx version in #535 (@fg-mindee)
  • Fixed TransformerDecoder for PyTorch 1.10 in #539 (@fg-mindee)
  • Fixed a bug in resolve_lines in #537 (@charlesmindee)
  • Fixed target computation in MASTER model (PyTorch backend) in #546 (@charlesmindee)
  • Fixed portuguese entry in VOCAB in #571 (@fmobrj)
  • Fixed header check typo in #557 (@fg-mindee)
  • Fixed keras version constraint in #579 (@fg-mindee)
  • Updated streamlit version in demo app in #611 (@charlesmindee)
  • Updated environment collection script in #575 (@fg-mindee)
  • Removed console print in builder in #566 (@fg-mindee)
  • Fixed docstring and export as xml dim bug in #586 (@felixdittrich92)
  • Fixed README instruction for page synthesis in #590 (@fg-mindee)
  • Adds missing console log and removed Tensorboard in #626 (@fg-mindee)
  • Fixed docstrings of datasets in #603 (@felixdittrich92)
  • Fixed documentation build requirements in #549 (@fg-mindee)

Improvements

  • Applied post release modifications in #520 (@fg-mindee)
  • Updated benchmark entry of crnn_mobilenet_v3 small in #523 (@charlesmindee)
  • Updated perf crnn_mobilenet_v3_large performances in doc (TF) in #526 (@charlesmindee)
  • Added automatic detection of rotated bbox in training utils in #534 (@fg-mindee)
  • Cleaned rotation transforms in #536 (@fg-mindee)
  • Updated library name spelling in #541 (@fg-mindee)
  • Updates README of detection training in #542 (@k-for-code)
  • Updated package index in #543 (@fg-mindee)
  • Updated README in #555 (@fg-mindee)
  • Updated CONTRIBUTING and issue templates in #560 (@fg-mindee)
  • Removed unused imports and prevents XML attacks in #582 (@fg-mindee)
  • Updated references to demo in README in #599 (@fg-mindee)
  • Updated readme and help in analyze.py in #596 (@RBMindee)
  • Specified that the API template only supports images for now in #609 (@fg-mindee)
  • Updated command to install tf/pytorch build in #614 (@charlesmindee)
  • Added checkpoint format to gitignore in #613 (@fg-mindee)
  • Specified comment in SAR about symbol encoding in #617 (@fg-mindee)
  • Drops support of np.float16 in #627 (@fg-mindee)

New Contributors

Our thanks & warm welcome to the following persons for their first contributions: @mzeidhassan @k-for-code @felixdittrich92 @SiddhantBahuguna @RBMindee @thentgesMindee :pray:

Full Changelog: https://github.com/mindee/doctr/compare/v0.4.0...v0.4.1

v0.4.0

2 years ago

This release brings the support of PyTorch out of beta, makes text recognition more robust, and provides light architectures for complex tasks.

Note: doctr 0.4.0 requires either TensorFlow 2.4.0 or PyTorch 1.8.0.

Highlights

No more width limitation for text recognition

Some documents such as French ID card include very long strings that can be challenging to transcribe:

fr_id_card_sample (copy)

This release enables a smart split/merge strategy for wide crops to avoid performance drops. Previously the whole crop was analyzed altogether, while right now, it is split into reasonably sized crops, the inference is performed in batch then predictions are merged together.

The following snippet:

from doctr.io import DocumentFile
from doctr.models import ocr_predictor

doc = DocumentFile.from_images('path/to/img.png')
predictor = ocr_predictor(pretrained=True)
print(predictor(doc).pages[0])

used to yield:

Page(
  dimensions=(447, 640)
  (blocks): [Block(
    (lines): [Line(
      (words): [
        Word(value='1XXXXXX', confidence=0.0023),
        Word(value='1XXXX', confidence=0.0018),
      ]
    )]
    (artefacts): []
  )]
)

and now yields:

Page(
  dimensions=(447, 640)
  (blocks): [Block(
    (lines): [Line(
      (words): [
        Word(value='IDFRABERTHIER<<<<<<<<<<<<<<<<<<<<<<', confidence=0.49),
        Word(value='8806923102858CORINNE<<<<<<<6512068F6', confidence=0.22),
      ]
    )]
    (artefacts): []
  )]
)

Framework specific predictors

PyTorch support is now no longer in beta, so we made some efforts so that switching from one deep learning backend to another is unified :raised_hands: Predictors are designed to be the recommended interface for inference with your models!

0.3.1 (TensorFlow) 0.3.1 (PyTorch) 0.4.0
>>> from doctr.models import detection_predictor
>>> predictor = detection_predictor(pretrained=True)
>>> out = predictor(doc, training=False)		 >>> from doctr.models import detection_predictor
>>> import torch
>>> predictor = detection_predictor(pretrained=True)
>>> predictor.model.eval()
>>> with torch.no_grad(): out = predictor(doc)		 >>> from doctr.models import detection_predictor
>>> predictor = detection_predictor(pretrained=True)
>>> out = predictor(doc)

An evergrowing model zoo :zebra:

As PyTorch goes out of beta, we have bridged the gap between PyTorch & TensorFlow pretrained models' availability. Additionally, by leveraging our integration of light backbones, this release comes with lighter architectures for text detection and text recognition:

  • db_mobilenet_v3_large
  • crnn_mobilenet_v3_small
  • crnn_mobilenet_v3_large

The full list of supported architectures is available :point_right: here

Demo live on HuggingFace Spaces

If you have enjoyed the Streamlit demo, but prefer not to run in on your own hardware, feel free to check out the online version on HuggingFace Spaces: Hugging Face Spaces

Courtesy of @osanseviero for deploying it, and HuggingFace for hosting & serving :pray:

Breaking changes

Deprecated crnn_resnet31 & sar_vgg16_bn

After going over some backbone compatibility and re-assessing whether all combinations should be trained, DocTR is focusing on reproducing the paper's authors' will or improve upon it. As such, we have deprecated the following recognition models (that had no pretrained params): crnn_resnet31, sar_vgg16_bn.

Deprecated models.export

Since doctr.models.export was specific to TensorFlow and it didn't bring much more value than TensorFlow tutorials, we added instructions in the documentation and deprecated the submodule.

New features

Datasets

Resources to access data in efficient ways

  • Added entry in vocabs for Portuguese #464 (@fmobrj), English, Spanish & German #467 (@fg-mindee), ancient Greek #500 (@fg-mindee)

IO

Features to manipulate input & outputs

  • Added .synthesize method to Page and Document #472 (@fg-mindee)

Models

Deep learning model building and inference

  • Add dynamic crop splitting for wide inputs to recognition models #465 (@charlesmindee)
  • Added MobileNets with rectangular pooling #483 (@fg-mindee)
  • Added pretrained params for db_mobilenet_v3_large #485 #487 , crnn_vgg16_bn #487, db_resnet50 #489, crnn_mobilenet_v3_small & crnn_mobilenet_v3_small #517 #516 (@charlesmindee)

Utils

Utility features relevant to the library use cases.

  • Added automatic font resolution function #472 (@fg-mindee)

Transforms

Data transformations operations

  • Added RandomCrop transformation #448 (@charlesmindee)

Test

Verifications of the package well-being before release

  • Added a unittest for RandomCrop #448 (@charlesmindee)
  • Added a unittest for crop split/merge in recognition models #465 (@charlesmindee)
  • Added unittests for PyTorch OCR model zoo #499 (@fg-mindee)

Documentation

Online resources for potential users

  • Added entry for RandomCrop #448 (@charlesmindee)
  • Added explanations about model export / compression #463 (@fg-mindee)
  • Added benchmark entry for db_mobilenet_v3_large #485 in the documentation (@charlesmindee)
  • Added badge with hyperlink to HuggingFace Spaces demo #501 (@osanseviero)

References

Reference training scripts

  • Added option to select vocab in the training of character classification and text recognition #502 (@fg-mindee)

Others

Other tools and implementations

  • Added CI job to validate the demo, the evaluation script and the environment collection scripts #456 (@fg-mindee), the character classification training script #457 (@fg-mindee), the analysis & evaluation scripts in PyTorch #458 (@fg-mindee), the text recognition scripts #469 (@fg-mindee), the text detection scripts #491 (@fg-mindee)
  • Added support of PyTorch for the analysis & evaluation scripts #458 (@fg-mindee)

Bug fixes

Datasets

  • Fixed submodule import #451 (@fg-mindee )
  • Added missing characters in French vocab #467 (@fg-mindee)

Models

  • Fixed PyTorch preprocessor shape resolution #453 (@charlesmindee)
  • Fixed Tensor cropping for channels_first format #458 #461 (@fg-mindee)
  • Replaced recognition models' MobileNet backbones by their rectangular pooling counterparts #483 (@fg-mindee)
  • Fixed crop extraction for PyTorch tensors #484 (@charlesmindee)
  • Fixed crop filtering on multi-page inference #497 (@fg-mindee)

Transforms

  • Fixed rounding errors in RandomCrop #473 (@fg-mindee)

Utils

  • Fixed page synthesis for characters outside of latin-1 #496 (@fg-mindee)

Documentation

  • Fixed READMEs of training scripts #504 #491 (@fg-mindee)

References

  • Fixed the requirements of the training scripts #494 #491 (@fg-mindee)

Others

  • Fixed the requirements of the streamlit demo #492 (@osanseviero), the API template #494 (@fg-mindee)

Improvements

Datasets

  • Merged DocDataset & OCRDataset #474 (@charlesmindee)
  • Updated DetectionDataset label format #491 (@fg-mindee)

Models

  • Deprecated doctr.models.export #463 (@fg-mindee)
  • Deprecated crnn_resnet31 & sar_vgg16_bn recognition models #468 (@fg-mindee)
  • Relocated DocumentBuilder to doctr.models.builder, split predictor into framework-specific objects #481 (@fg-mindee)
  • Added more robust argument checks in DocumentBuilder & refactored crop preparation and result processing in ocr predictors #497 (@fg-mindee)
  • Reflected changes of detection target formats on detection models #491 (@fg-mindee)

Utils

  • Improved page synthesis with dynamic font size #472 (@fg-mindee)

Documentation

  • Updated README badge & added release-specific documentation index #451 (@fg-mindee)
  • Added logo in README & documentation #459 (@charlesmindee)
  • Updated hyperlink to documentation in the README #462 (@fg-mindee)
  • Updated vocab description in the documentation #467 (@fg-mindee)
  • Added favicon in the documentation #466 (@fg-mindee)
  • Removed benchmark entry of deprecated models #468 (@fg-mindee)
  • Updated README of the text recognition training script #469 (@fg-mindee)
  • Updated performance benchmark with crop splitting #471 (@charlesmindee)
  • Added page synthesis example in README #472 (@fg-mindee)
  • Made copyright mention dynamic, improved the landing & installation pages in the documentation #475 (@fg-mindee)
  • Restructured the documentation #519 (@fg-mindee)

Tests

  • Removed legacy unittests of doctr.models.export #463 (@fg-mindee)
  • Removed unittests for deprecated models #468 (@fg-mindee)
  • Updated unittests with the new doctr.utils.font submodule #472 (@fg-mindee)
  • Reflected changes from predictor refactor #481 (@fg-mindee)
  • Extended unittest of crop extraction #484 (@charlesmindee)
  • Reflected changes from predictor crop preparation improvement #497 (@fg-mindee)
  • Reflect changes from detection target format #491 (@fg-mindee)

References

  • Reflected changes of detection dataset target format #491 (@fg-mindee)

Others

  • Specified import of file_utils #447 (@zalakbhalani)
  • Updated package version #451 (@fg-mindee)
  • Updated PIL version constraint to fix vulnerability #460 (@fg-mindee)
  • Updated model selection in the demo #468 (@fg-mindee)
  • Removed some MacOS CI jobs that were slowing down PR checks #470 (@fg-mindee)
  • Reflected page synthesis changes in demo #477 (@fg-mindee)
  • Reflected changes from predictor refactor in API & demo #481 (@fg-mindee)
  • Updated author_email in setup #493 (@fg-mindee)
  • Split CI jobs for pytest in common, pytorch & tensorflow #498 #503 #506 (@fg-mindee)
  • Removed unused imports #507 (@fg-mindee)

Many thanks to our contributors, we are delighted to see that there are more every week!

v0.3.1

2 years ago

This release stabilizes the support for PyTorch backend while extending the range features (new task, superior pretrained models, speed ups).

Brought to you by @fg-mindee & @charlesmindee

Note: doctr 0.3.1 requires either TensorFlow 2.4.0 or PyTorch 1.8.0.

Highlights

Improved pretrained parameters for your favorite models :rocket:

Which each release, we hope to bring you improved models and more comprehensive evaluation results. As part of the 0.3.1 release, we provide you with:

  • improved params for crnn_vgg16_bn & sar_resnet31
  • evaluation results on a new private dataset (US tax forms)

Lighter backbones for faster architectures :zap:

Without any surprise, just like many other libraries, DocTR's future will involve some balance between speed and pure performance. To make this choice available to you, we added support of MobileNet V3 and pretrained it for character classification for both PyTorch & TensorFlow.

Speeding up preprocessors & datasets :train2:

Whether you are a user looking for inference speed, or a dedicated model trainer looking for optimal data loading, you will be thrilled to know that we have greatly improved our data loading/processing by leveraging multi-threading!

Better demo app :art:

We value the accessibility of this project and thus commit to improving tools for entry-level users. Deploying a demo from a Python library is not the expertise of every developer, so this release improves the existing demo:

new_demo

Page selection was added for multi-page documents, the predictions are used to produce a synthesized version of the initial document, and you get the JSON export! We're looking forward to your feedback :hugs:

[beta] Character classification

As DocTR continues to move forward with more complex tasks, paving the way for a consistent training procedure will become necessary. Pretraining has shown potential in many deep learning tasks, and we want to explore opportunities to make training for OCR even more accessible. char_classif

So this release makes a big step forward by adding on-the-fly character generator and training scripts, which allows you to train a character classifier without any pre-existing data :hushed:

Breaking changes

Default dtype of TF datasets

In order to harmonize data processing between frameworks, the default data type of dataloaders has been switched to float32 for TensorFlow backend:

0.3.0 0.3.1
>>> from doctr.datasets import FUNSD
>>> ds = FUNSD()
>>> img, target = ds[0]
>>> print(img.dtype)
<dtype: 'uint8'>
>>> print(img.numpy().min(), img.numpy().max())
0 255		 >>> from doctr.datasets import FUNSD
>>> ds = FUNSD()
>>> img, target = ds[0]
>>> print(img.dtype)
<dtype: 'float32'>
>>> print(img.numpy().min(), img.numpy().max())
0.0 1.0

I/O module

Whether it is for exporting predictions or loading input data, the library lets you play around with inputs and outputs using minimal code. Since its usage is constantly expanding, the doctr.documents module was repurposed into doctr.io.

0.3.0 0.3.1
>>> from doctr.documents import DocumentFile
>>> pdf_doc = DocumentFile.from_pdf("path/to/your/doc.pdf").as_images()		 >>> from doctr.io import DocumentFile
>>> pdf_doc = DocumentFile.from_pdf("path/to/your/doc.pdf").as_images()

It now also includes an image submodule for easy tensor <--> numpy conversion for all supported data types.

Multithreading relocated

As multithreading is getting increasingly used to boost performances in the entire library, it has been moved from utilities of TF-only datasets to doctr.utils.multithreading:

0.3.0 0.3.1
>>> from doctr.datasets.multithreading import multithread_exec
>>> results = multithread_exec(lambda x: x ** 2, [1, 4, 8])		 >>> from doctr.utils.multithreading import multithread_exec
>>> results = multithread_exec(lambda x: x ** 2, [1, 4, 8])

New features

Datasets

Resources to access data in efficient ways

  • Added support of FP16 (#367)
  • Added option to merge subsets for recognition datasets (#376)
  • Added dynamic sequence encoding (#393)
  • Added support of new label format datasets (#407)
  • Added character generator dataset for image classification (#412, #418)

IO

Features to manipulate input & outputs

  • Added Element creation from dictionary (#386)
  • Added byte decoding function for PyTorch and TF (#390)
  • Added extra tensor conversion functions (#412)

Models

Deep learning model building and inference

  • Added crnn_resnet31 as a recognition model (#361)
  • Added a uniform comprehensive preprocessing mechanism for both frameworks (#370)
  • Added support of FP16 (#382)
  • Added MobileNet V3 for TensorFlow as a backbone (#372, #410, #420)
  • Added superior pretrained params for crnn_vgg16_bn in TF (#395)
  • Added pretrained params for master in TF (#396)
  • Added mobilenet backbone availability to detection & recognition models (#398, #399)
  • Added pretrained params for mobilenets on character classification (#415, #421, #424)
  • Added superior pretrained params for sar_resnet31 in TF (#395)

Utils

Utility features relevant to the library use cases.

  • Added box rotation function (#358)
  • Added box visualization feature (#384)

Transforms

Data transformations operations

  • Added rotate function (#358) and its corresponding augmentation module (#363)
  • Added cropping function (#366)
  • Added support of FP16 (#388)

Test

Verifications of the package well-being before release

  • Added unittests for rotation functions (#358)
  • Added test cases for recognition zoo (#361)
  • Added unittests for RandomRotate (#363)
  • Added test cases for recognition dataset merging (#376)
  • Added unittests for Element creation from dicts (#386)
  • Added test case for mobilenet backbone (#372)
  • Added unittests for datasets with new format (#407)
  • Added test cases for the character generator (#412)

Documentation

Online resources for potential users

  • Added entry for RandomRotate (#363)
  • Added entry for CharacterGenerator (#412)
  • Added evaluation on US tax forms in the documentation (#419)

References

Reference training scripts

  • Added PyTorch training reference scripts (#359, #394)
  • Added LR scheduler to TensorFlow script (#360, #374, #397) & Pytorch scripts (#381)
  • Added possibility to use multi-folder datasets (#377)
  • Added character classification training script (#414, #420)

Others

Other tools and implementations

  • Added page selection and result JSON display in demo (#369)
  • Added an entry for MASTER in model selection of the demo (#400)

Bug fixes

Datasets

  • Fixed image shape resolution in custom datasets (#354)
  • Fixed box clipping to avoid rounding errors in datasets (#355)

Models

  • Fixed GPU compatibility of detection models (#359) & recognition models (#361)
  • Fixed recognition model loss computation in PyTorch (#379)
  • Fixed loss computation of CRNN in PyTorch (#434)
  • Fixed loss computation of MASTER in PyTorch (#440)

Transforms

  • Fixed Resize transformation when aspect ratio matches the target (#357)
  • Fixed box rotation (#378)
  • Fixed image expansion while rotating (#438)

Documentation

  • Fixed installation instructions (#437)

References

  • Fixed missing import in utils (#389)
  • Fixed GPU support for PyTorch in the recognition script (#427)
  • Fixed gradient clipping in Pytorch scripts (#432)

Others

  • Fixed trigger of script testing CI job (#351)
  • Constrained PIL version due to issues with version 8.3 (#362)
  • Added missing mypy config ignore (#365)
  • Fixed PDF page rendering for demo & analysis script (#368)
  • Constrained weasyprint version due to issues with version 53.0 (#404)
  • Constrained matplotlib version due to issues with version 3.4.3 (#413)

Improvements

Datasets

  • Improved typing of doctr.datasets (#354)
  • Improved PyTorch data loading (#362)
  • Switched default dtype of TF to tf.float32 instead of tf.uint8 (#367, #375)
  • Optimized sequence encoding throught multithreading (#393)

IO

  • Relocated doctr.documents to doctr.io (#390)

Models

  • Updated bottleneck channel multiplier in MASTER (#350)
  • Added dropout in MASTER (#349)
  • Renamed MASTER attribute for consistency across models (#356)
  • Added target validation for detection models (#355)
  • Optimized preprocessing by leveraging multithreading (#370)
  • Added dtype dynamic resolution and specified error messages (#382)
  • Harmonized parameter loading in PyTorch (#425)
  • Enabled backbone pretraining in complex architectures (#435)
  • Made head & FPN sizing dynamic using the feature extractor in detection models (#435)

Utils

  • Moved multithreading to doctr.utils (#371)
  • Improved format validation for visualization features (#392)
  • Moved doctr.models._utils.rotate_page to doctr.utils.geometry.rotate_image (#371)

Documentation

  • Updated pypi badge & documentation changelog (#346)
  • Added export example in documentation (#348)
  • Reflected relocation of doctr.documents to doctr.io in documentation and README (#390)
  • Updated recognition benchmark (#395, #441)
  • Updated model entries (#435)
  • Updated authors & maintainer references in setup.py and in README (#444)

Tests

  • Added test case for same aspect ratio resizing (#357)
  • Extended testing of datasets (#354)
  • Added test cases for FP16 support of datasets (#367), models (#382), transforms (#388)
  • Moved multithreading unittests (#371)
  • Extended test cases of preprocessors (#370)
  • Reflected relocation of doctr.documents to doctr.io (#390)
  • Removed unused imports (#391)
  • Extended test cases for visualization (#392)
  • Updated unittests of sequence encoding (#393)
  • Added extra test cases for rotation validation (#438)

References

  • Added optimal selection of workers for all scripts (#362)
  • Reflected changes of switching to tf.float32 by default for datasets (#367)
  • Removed legacy script arg (#380)
  • Removed unused imports (#391)
  • Improved device selection for Pytorch training script (#427)
  • Improved metric logging when the value is undefined (#432)

Others

  • Updated package version (#346)

v0.3.0

2 years ago

This release adds support for PyTorch backend & rotated text elements.

Release brought to you by @fg-mindee & @charlesmindee

Note: doctr 0.3.0 requires either TensorFlow 2.4.0 or PyTorch 1.8.0.

Highlights

[beta] Welcome PyTorch :tada:

This release comes with exciting news: we added support of PyTorch for the whole library!

If you have both TensorFlow & Pytorch, simply switch DocTR backend by using the USE_TORCH and USE_TF environment variables.

export USE_TORCH='1'

Then DocTR will do the rest for you to play along with PyTorch:

import torch
from doctr.models import db_resnet50
model = db_resnet50(pretrained=True).eval()
with torch.no_grad():
    out = model(torch.rand(1, 3, 1024, 1024))

More pretrained models to come in the next releases!

Support of rotated boxes

Users might be tempted to filtered text recognition predictions, which was not easy previously without a prediction's confidence. We harmonized our recognition models to provide the sequence prediction probability.

Rotated bounding boxes

Page reconstruction

Following up on some feedback about the lack of clarity for visualization of dense predictions, we added a page reconstruction feature.

import matplotlib.pyplot as plt
from doctr.utils.visualization import synthesize_page
from doctr.documents import DocumentFile
from doctr.models import ocr_predictor

model = ocr_predictor(pretrained=True)
# PDF
doc = DocumentFile.from_pdf("path/to/your/doc.pdf").as_images()
# Analyze
result = model(doc)

# Reconstruct the first page
reconstructed_page = synthesize_page(result.export()[0])
plt.imshow(reconstructed_page); plt.show()

Original image Page reconstruction

Using the predictions from our models, we try to synthesize the document with only its textual information!

Breaking changes

Renamed LinkNet

While the paper doesn't introduce different versions of the LinkNet architectures, we want to keep the possibility to add more. In order to stabilize the interface early on, we renamed linknet into linknet16

0.2.1 0.3.0
>>> from doctr.models import linknet
>>> model = linknet(pretrained=True)		 >>> from doctr.models import linknet16
>>> model = linknet16(pretrained=True)

New features

Datasets

Resources to access data in efficient ways

  • Added option to yield rotated bounding boxes as target (#281)
  • Added support of PyTorch for all datasets (#319)

Documents

Features to manipulate document information

  • Added support of rotated bboxes (#281)
  • Added entry for MASTER (#300)
  • Updated LinkNet entry (#313)
  • Added code of conduct (#325)

Models

Deep learning model building and inference

  • Added rotated cropping feature & inference mode (#281)
  • Added spatial masked loss support for LinkNet (#296)
  • Added page orientation estimation feature (#293)
  • Added box target rotation feature (#297)
  • Added support of MASTER recognition model & transformer (#300, #342)
  • Added Focal loss support to linknet (#304, #311)
  • Added PyTorch support for DBNet (#310, #313, #316), LinkNet (#317), conv_sequence & parameter loading (#323), resnet31 (#327), vgg16_bn (#328), CRNN (#318), SAR (#333), MASTER (#329, #335, #340, #342)
  • Added cleaner verified file downloading function (#319)
  • Added upfront page orientation estimation (#324) by @Rob192

Utils

Utility features relevant to the library use cases.

  • Added Mask IoU computation (#290)
  • Added straight <--> rotated bbox conversion and metric computation support (#281)
  • Added page synthesis feature (#320)
  • Added IoA, and NMS (#332)

Transforms

Data transformations operations

  • Added support of custom Resize in PyTorch (#313), ColorInversion (#322)

Test

Verifications of the package well-being before release

  • Added unittest for maks IoU computation (#290)
  • Added unittests for rotated bbox support (#281, #297)
  • Added unittests for page orientation estimation (#293, #324)
  • Added unittests for MASTER (#300, #309)
  • Added test case for the focal loss of LinkNet (#304)
  • Added unittests for Pytorch integration (#310, #313, #317, #319, #322, #323, #327, #318, #329, #335, #340, #342)
  • Added unittests for IoA & NMS (#332)

Documentation

Online resources for potential users

  • Added instructions to install DocTR with PyTorch or TF (#306)
  • Added specific instructions to run checks in CONTRIBUTING (#321)

References

Reference training scripts

  • Added support of rotated bounding box targets (#281)

Others

Other tools and implementations

  • Added support of rotated bounding box target & inference mode (#281)
  • Added framework availability check (#306, #314, #315)
  • Added CI job for pytorch unittests (#310)
  • Added CI jobs to build DocTR with multiple python version, environment and framework (#314, #315)
  • Updated demo to add page reconstruction (#320)
  • Added PyTorch & torchvision to environment collection script (#345) & updated the bug template

Bug fixes

Documentation

  • Fixed entry of datasets (#344)

Tests

  • Fixed ColorInversion unittest (#298, #339)

References

  • Fixed missing import of wandb in the detection script (#288)
  • Fixed edge case of recognition model output unpacking in the recognition training script (#291)
  • Fixed model output unpacking in the detection script (#301)
  • Fixed wandb config for training scripts (#302)

Others

  • Fixed edge case of recognition model output unpacking in the evaluation script (#291)
  • Fixed mypy config and related typing annotations (#308, #312, #314, #336)

Improvements

Datasets

  • Improved constructors of OCRDataset and CORD (#289, #299)
  • Silenced numpy dtype warnings (#336)

Documents

  • Updated README badge & documentation versioning (#287)
  • Harmonized benchmark table formatting of figures (#281)
  • Updated demo illustration in README (#326)

Documentation

  • Updated documentation font and mentioned PyTorch support in README & docs (#344)

Tests

  • Updated unittest image (#337)
  • Cleaned up unittest folder separation (#338)

References

  • Reordered script option to save time for test-only (#294)

Others

  • Updated package version (#287)
  • Removed unused imports (#295, #307, #336)
  • Updated API requirements for security and cleaned Dockerfile (#303)
  • Improved setuptools classifiers and installation process (#306)

:pray: Thanks to our contributors :pray: @Rob192