Azure Form Recognizer

#Azure #FormRecognizer #AIBuilder #PowerApps #CanvasApps

Extract & Store data from Documents with AI Builder with Power Apps

Photo by Towfiqu barbhuiya on Unsplash

Azure Form Recognizer is a powerful tool that allows businesses to automate their data collection process and gain actionable insights from forms and documents. This cloud-based service provided by Microsoft is built on the latest artificial intelligence (AI) technologies, including optical character recognition (OCR) and natural language processing (NLP).

With Azure Form Recognizer, businesses can easily extract data from forms and documents, such as invoices, receipts, contracts, and more. The service can also recognize handwriting, making it suitable for extracting information from handwritten forms. One of the key features of Azure Form Recognizer is its ability to learn from examples. This means that businesses can train the service to recognize specific forms and extract the relevant information.

Scan & Extract Receipt Documents

In this article we will discuss about scanning receipt documents with PowerApps AI Builder.

PowerApps AI Builder

PowerApps AI Builder is a tool allows users to easily create & integrate artificial intelligence (AI) models into their PowerApps applications. This tool is built on top of the Microsoft Power Platform and allows users to easily create custom AI models without needing to have a deep understanding of machine learning or data science.

With PowerApps AI Builder, users can create AI models that can be used to perform tasks such as image and text recognition, sentiment analysis, and form processing. One of the key features of PowerApps AI Builder is its ability to use pre-built models.. Instead, they can use pre-built models that have been trained on large amounts of data and fine-tuned for specific tasks.

AI Builder & Form Recognizer

AI Builder document processing is built on top of Azure Form Recognizer. This provides both products with the latest advancements in Microsoft AI.

AI Builder is part of Microsoft Power Platform. This enables anyone to add AI into apps and automation with an easy-to-use interface. You don’t need to be a developer or data scientist.

Azure Form Recognizer is targeted to professional developers. They can use simple REST APIs to add AI capabilities to their custom code solutions.

Ref: Frequently asked questions (FAQ) for document processing — AI Builder | Microsoft Learn

How to use AI Model in Power Apps

Navigate to make.powerapps.com and select your environment for building this application

Image Ref from : https://powerapps.microsoft.com/en-us/ai-builder/

Using Prebuilt model

Ref Image: https://powerapps.microsoft.com/en-us/blog/process-receipts-with-ai-builder/

We can upload different receipts to test extraction with prebuilt model. Following prebuilt model can be used in Automate flow or canvas app for usage in case of prebuilt model solves your business scenario.

Using AI Builder in Canvas Apps

Custom Modeling

In case of Prebuilt model is not extracting right information we can create custom models to train documents and create model to consume in AI Builder.

Step 1

Choose document type

Step 2

Choose information to extract

Step 3

Add collections of documents

Step 4

Tag documents

Step 5

Model summary

Create Custom Labels to Extract from documents

Receipt Total

Tag Labels from uploaded documents

Labelling

After labelling/tagging it will be ready for training. After training it will be ready to use in AI Builder

Extract & Store information from Documents with Power Automate

Power automate has an action to extract information from receipts

Image Ref: https://learn.microsoft.com/en-us/training/modules/ai-builder-power-automate/3-ai-builder-actions

Image Ref : https://learn.microsoft.com/en-us/ai-builder/flow-receipt-processing

Extracted information from this step can be stored in Excel/Dataverse or Azure Table Storage with different connectors

With AI Builder ,it is easy to use prebuilt models or creating custom models & integrate with Power apps.

As discussed earlier if you are a professional developer (with basic understanding of Azure/Rest API) ,then there will be more options with Azure Form Recognizer .

Lets explore Form Recognizer…

Create Model with Form Recognizer

Step 1

Create Azure Cognitive Service Resource

Step 2

Create Azure Storage Account to store documents for Training

Step 3

Create Labelling with Form Recognizer Studio

Step 4

Train(Template v/s Neural Models) & Validate Accuracy

Step 5

Consume Form Recognizer with Custom Azure Function & Power Automate

Step 1

Navigate to portal.azure.com

Ref: https://azure.microsoft.com/en-us/products/cognitive-services/#overview

Ref: https://learn.microsoft.com/en-us/azure/cognitive-services/cognitive-services-apis-create-account?tabs=multiservice%2Canomaly-detector%2Clanguage-service%2Ccomputer-vision%2Cwindows

Image Ref: Microsoft Docs

Step 2: Create Azure Storage Account to store documents for Training

Create Azure Storage account to store documents required for Training.

This storage account will be one of the parameter for documents labelling as source.

Step 3: Create Azure Storage Account to store documents for Training

Navigate to https://formrecognizer.appliedai.azure.com/

Image Ref: Form Recognizer Home Page

within form recognizer studio we can use consume prebuilt models to create models to use or generate custom model with labelling

it is very similar experience with tagging/labelling in AI builder as discussed earlier.

Step 4: Template V/s Neural

After labelling next step will be creating custom model

There is a difference with experience when compared to AI builder.In Form recognizer we will have two options to choose for modeling

1. Template
2. Neural

As per official documentation

Template models work well when the target documents share a common visual layout. Training only takes a few minutes, and more than 100 languages are supported.

Neural models can flexibly handle both structured and unstructured documents. Training takes up to half an hour, and currently only English language documents are supported. The current version can extract inline field data and checkboxes.
Neural models are available only in select regions. Click here for details.

Step 5: Consume Form Recognizer with Custom Azure Function & Power Automate

We can consume Custom model with “Azure.AI.FormRecognizer.DocumentAnalysis” SDK

DocumentAnalysisClient Class

(Azure.AI.FormRecognizer.DocumentAnalysis) — Azure for .NET Developers | Microsoft Learn

As per documentation

The client to use to connect to the Form Recognizer Azure Cognitive Service to analyze information from documents and images and extract it into structured data.

Main Properties in the response to retrieve data

AnalyzeResult result : Response Type

result.Documents : Contains Extracted Information

document.Fields :Contains Labeled fields

With Power Automate

As per documentation

The Form Recognizer connector provide integration to Cognitive Service Form Recognizer. It allows analyze and extract information from Forms, Invoices, Receipts, Business Cards, and ID Documents.

Form Recognizer - Connectors

Ref: Power Automate

Table Extraction

Table extraction is also major/required feature in extracting information.

Table labelling supported in both AI Builder & Form Recognizer Studio.

With December 2022 update

Auto table labeling. After you select the table icon within a document, you can opt to auto-label the extracted table in the labeling view.

Label subtypes and second-level subtypes The Studio now supports subtypes for table columns, table rows, and second-level subtypes for types such as dates and numbers.

https://learn.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/whats-new?view=form-recog-3.0.0&tabs=csharp

I hope this will help you to start creating your own app to scan receipts & store, analyze your expenses

Photo by Claudio Schwarz on Unsplash

Just imagine having your own Power BI /Excel report with all your restaurant expenses :) :) :)

Photo by Lukas Blazek on Unsplash

Help links

What is Azure Form Recognizer - Azure Applied AI Services

Get started with Power Automate (contains video) - Power Automate

Quickstart: Label forms, train a model, and analyze forms using the Sample Labeling tool - Form…