A demo of Azure Form Recognizer (Custom Model) with Azure Function blob trigger to process, tag, and move a patient referral form, then save extracted data to Azure Table Storage.
The purpose of this repository is to demonstrate the use of Azure Form Recognizer (v3) in evaluating a patient referral form sent from doctors and hospitals. It is assumed that doctors will attach a standardized coversheet when they refer a patient.
Forms are used to communicate information in every industry, every day. Many people still manually extract data from forms to exchange information. When doctors refer patients to hospitals, they frequently Fax (yes, many still do) the referral to the hospital for administrative purposes. This calls for a series of employees to review the documents upon arrival and enter its details into the system correctly - a laborious task that is prone to human error.
Each health provider has their own style of referral document and while intent and patient data fields are generally present, sometimes information is missing from these highly individual documents. To solve this, one state health authority is now providing a standardised coversheet to be attached to the fax referral, mandating which attributes must be supplied.
Our goal is now to train an Azure Form Recognizer model to label, train, and test the optical character recognition to detect the values supplied in these referral forms, minimising the cognitive effort by frontline workers.
incoming
container of the Storage Account, it is picked up for processing by the Azure Function (which listens for changes to blobs in this containers).review
container for manual / human downstream processing.completed
container.incoming
(source) container.BlobTrigger
- when a blob in a storage container changes, such as when it is added.In this interactive process, you tell Form Recognizer what text to extract from the coversheet, based on your training dataset of at least 5 images. You may use the Form Recognizer Studio experience to upload a set of coversheet documents (supplied) with different values for the fields (e.g. fields, selection marks, signatures, and tables).
training
Project Details | Service Resources | Training Source | Review |
---|---|---|---|
If you are seeing the following CORS error after opening the Form Recognizer project, please ensure you follow this guide to configure CORS. Please note that the origin URL must not contain a trailing /
character.
training
container).Peter
Pan
then you can select both and mark them as the name).Initially: Unlabelled Training Dataset | 1. Labelling Selection Marks | 2. Labelling Text Fields | 3. Document Regions / Signatures |
---|---|---|---|
If your text fields are dates or integers for example, you may wish to update the sub type of these fields as follows:
This occurs because you may have selected more than one snippet of text when the chosen field type (ie. selection marks or signatures) only support one. Simply click out of the document and back into it to clear your selection and try again.
⚠️ Please remember that (at the time of writing this) least 5 OCR files are required to train a model. A 'ModelBuildError' will appear if you attempt to train with less than 5 files.
Once you have trained a model, you can test it:
+ Add
button🎉 Congratulations! You have successfully deployed, labeled, trained, and tested an Azure Form Recognizer Custom Form!
In this chapter, we are looking at implementing the Azure Function code for the Azure Function App you deployed in chapter 1. Please note that Azure Functions supports a range of common programming languages - documented here.
It is also possible to use "no/low-code" tools including Azure Logic Apps or Power Automate for this scenario. However, this example shows the use of a Blob-triggered Azure Function with the TypeScript language.
It is assumed you have basic coding knowledge and are familiar with the concept of CI/CD to deploy the Function App code with version control in mind.
{
"MIN_CONFIDENCE_SCORE": 0.8, // Minimum acceptable confidence (between 0-1)
"FORM_RECOGNIZER_STORAGE": "DefaultEndpointsProtocol=https;AccountName=[...];AccountKey=[...];EndpointSuffix=core.windows.net",
"FORM_RECOGNIZER_API_KEY": "[...]",
"FORM_RECOGNIZER_MODEL_ID": "[...]",
"FORM_RECOGNIZER_ENDPOINT": "https://[...].cognitiveservices.azure.com"
}
ℹ️ To locate the Storage Account connection string for
FORM_RECOGNIZER_STORAGE
, navigate to your Storage Account in the Azure Portal, click on Access Keys and copy the Connection String value of either key1 or key2.
Now that your function code is deployed. It whenever a new blob is added into the Storage Account container, the function is triggered and begins processing the blob. As per the Function App code, the output will be saved into Azure Table Storage, a cheap, and extremely fast key-value pair store. You may decide to switch this out for a SQL Database to meet your operational needs.
If you have made it this far, you will see that your detected fields have been saved into Azure Table Storage as per out architecture diagram.
Additionally, the processed file - with confidence metadata added - is moved from the incoming
container to either the processed
or review
container depending on whether the confidence score is greater than or less than the MIN_CONFIDENCE_SCORE
threshold defined in our environment variables:
In the real scenario that you have multiple instances of Azure Form Recognizer for dev/test and prod for instance, you may wish to copy a model you trained in your dev/test environment into the production environment.
✨ Tip: The Form Recognizer Studio (web interface) does not currently support this feature. However, it is simple to use the API in this scenario, or download the API Definition and import it into Postman or other HTTP clients.
Pre-requisites: We need to collect the following four pieces of information:
We first Generate a Copy Authorization. This generates an authorization key we can user to copy a model to this location with specified model id and optional description.
Ocp-Apim-Subscription-Key
: Must contain the key of the destination Form Recognizer instance, in this case the Production Form Recognizer Key (copied earlier).Content-Type
: Must be set to application/json
We supply the name that we would like to give our model upon transfer, and an optional description.
{
"modelId": "TransferredModel",
"description": "This is a description for our copied model, as it will appear in the target resource."
}
POST https://australiaeast.api.cognitive.microsoft.com/formrecognizer/documentModels:authorizeCopy?api-version=2021-09-30-preview HTTP/1.1
Host: australiaeast.api.cognitive.microsoft.com
Content-Type: application/json
Ocp-Apim-Subscription-Key: ••••••••••••••••••••••••••••••••
{
"modelId": "TransferredModel",
"description": "This is a description for our copied model, as it will appear in the target resource."
}
We need to copy the response body for the next step.
...
{
"targetResourceId": "/subscriptions/[subscription_id]/resourceGroups/[resource_group]/providers/Microsoft.CognitiveServices/accounts/[prod-form-recognizer-name]",
"targetResourceRegion": "australiaeast",
"targetModelId": "TransferredModel",
"targetModelLocation": "https://australiaeast.api.cognitive.microsoft.com/formrecognizer/documentModels/TransferredModel?api-version=2021-09-30-preview",
"accessToken": "ffd3e3f9-e9de-4e9c-b29d-4ac16bdf4f3e",
"expirationDateTime": "2022-02-01T09:58:50Z"
}
Now that we have an authorization token in hand, we can invoke the Copy Model API, which copies the model to the target (production) resource.
modelId
: Must contain the ID of the model we wish to transfer (i.e. the ID of the model we trained in Chapter 2 and now wish to copy to the production environment).Ocp-Apim-Subscription-Key
: Must contain the key of the source Form Recognizer instance, in this case the Dev/Test Form Recognizer Key (copied earlier).Content-Type
: Must be set to application/json
We simply copy the response body from the previous step, which contains the authorization token and resource information.
{
"targetResourceId": "/subscriptions/[subscription_id]/resourceGroups/[resource_group]/providers/Microsoft.CognitiveServices/accounts/[prod-form-recognizer-name]",
"targetResourceRegion": "australiaeast",
"targetModelId": "TransferredModel",
"targetModelLocation": "https://australiaeast.api.cognitive.microsoft.com/formrecognizer/documentModels/TransferredModel?api-version=2021-09-30-preview",
"accessToken": "ffd3e3f9-e9de-4e9c-b29d-4ac16bdf4f3e",
"expirationDateTime": "2022-02-01T09:58:50Z"
}
When the request is made, you should receive a 202 Accepted
status code. This indicates that the model will now be copied in the background. For error codes and up-to-date information, please refer to the Cognitive Services API Documentation
🎉 Congratulations, you have successfully copied the model from one instance to another.
Azure Form Recognizer is just one of multiple Applied AI services available in the Azure cloud. The implementation of this service repends on the scenario and (for demo purposes) does not currently account for additional security requirements your organization may have.
If you found this resource helpful, feel free to connect with me on LinkedIn or make my day by buying me a coffee to keep fuelling projects like these.
Best regards, Olaf Wrieden