In this directory, you can find several notebooks that illustrate how to use Microsoft's TrOCR both for fine-tuning on custom data as well as inference. It currently includes the following notebooks:
- performing inference with TrOCR to illustrate optical character recognition with Transformers, as well as making a Gradio demo
- fine-tuning TrOCR on the IAM dataset using HuggingFace's Seq2SeqTrainer
- fine-tuning TrOCR on the IAM dataset using native PyTorch
I also made a notebook that illustrates how to evaluate a TrOCR checkpoint in terms of CER (character-error rate) on the test set of the IAM Handwriting Database.
All models can be found on the hub.
Note that there's also a Gradio demo available for TrOCR, hosted as a HuggingFace Space here.
There's a nice blog post on reducing the character error rate (CER) by training TrOCR with a margin loss instead of the cross-entropy loss: https://blog.doxray.com/p/generative-trocr-models-calibration.