Fields and extraction

Fields are the values you want from each document: an invoice number, a date, a total, a supplier, a table of line items. DocView reads them with OCR-based data extraction, checks them, shows them to operators when needed, and writes them to your data files and systems.

Fields belong to a document class. You find them under Fields on the Document classes & fields tab, below the class's classification rules. Batch fields, which apply to the whole batch, are on the General tab and work the same way.

Add a field

1. Open the Batch Setup, go to Document classes & fields and click the document class. Create a draft first if the version is released.
2. Under Fields, click Add field.
3. Fill in the Basics (see below).
4. Choose how the value is found under Extraction.
5. Add checks under Formatters and validators and Validation.
6. Click Save field.

Drag fields by the grip, or use the arrows, to set their order on the operator screens. The field list shows each field's type, extraction method or finder, confidence threshold and flags such as required, read-only, hidden, double-key and script.

Switch on Advanced options at the top of the field editor to see the rarely used settings, such as the calculation script.

Basics

Extraction methods

Under Extraction you choose where the value comes from.

• Value from finder takes the value from a finder of the document class. Finders are the most powerful option: they can search near each other, fall back to each other and combine results. See Finders and zones. A finder replaces the field's own method.
• Otherwise choose a Method and fill in its Settings.

Regular expression tips

Put brackets around the part you want and set Part to use to 1. For example, to read an order number:

Order\s*No\.?\s*:?\s*(\d+)

Part to use 0 takes the whole match. The pattern box offers a quick test so you can try it on sample text.

Key-value tips

List several labels, most specific first. For a total, list Total due, Amount due, Invoice total and Total, and switch on Use the last occurrence. That way a "Total" in the line items does not win over the grand total at the bottom.

Zones

For Zonal OCR, Anchor-relative zone and Tick boxes (OMR), you rarely type numbers. Save the field, then click Open the zone designer under the settings and draw the zone, or the tick boxes, on a sample. See Finders and zones.

Where the value comes from

Beside the field form, the panel Where the value comes from shows a sample document of the Batch Setup: the newest sample of the field's document type. It marks where the value is read:

• the field's own zone or tick boxes,
• the zone or tick boxes of its finder, or its zone of an Advanced zones finder,
• or the finder's search areas (all of them).

The panel follows the form while you edit. Click Test on sample to read the value from the sample. Samples are kept with the Batch Setup: upload them in the zone designer or in the Samples & testing panel of the Finders tab, or keep them from an import.

Forms and tick boxes

Zonal OCR has optional settings for forms. register makes the zone follow printed boxes near it, and pad adds a few millimetres around it. For hand printing in comb boxes, dropout, psm, whitelist, upper and comb keep only the pen strokes and join the letters. Tick boxes (OMR) reads a group of boxes into one value, or tells you whether a signature box is signed. To read many zones of a form at once, use an Advanced zones (form) finder and take each field's value from one of its zones, for example AZL.Surname. See Forms, tick boxes and signatures.

Tables and line items

Choose the Table data type for line items. After saving, add the columns under Table columns with Add column. Each column is a field with its own data type, and Column width (% of the table) sets its default width on the Index screen.

For a table field, the Extraction section only shows when Advanced options is on. Choose the Table / line items method there. In its settings, map each column to the header printed on the document. Separate alternative headers with |, for example Qty|Quantity|Hours. Leave Columns empty to use the table field's own columns. Reading stops at lines starting with the words in Stop at lines starting with, such as Subtotal and Total.

Confidence threshold

Confidence threshold (0 to 1) decides when a value needs a human look. Values extracted below it are flagged for review in Correction and Index / Validate. The default is 0.8. Lower it for fields that are usually right; raise it for fields where a mistake is costly.

Database lookups

A field can use a Database lookup to check its value against your own data, fill other fields from the matching row, or suggest values while the operator types. Under Lookup fills these fields, map lookup columns to fields, one per line:

Name = Supplier
VatNumber = VatNumber

Columns named like a field fill it anyway. See Lookups.

Formatters and validators

Besides the simple formatters (trim, case, remove characters, find and replace, prefix, pad…), three handle what OCR and different suppliers do to values:

Formatters tidy a value, for example removing spaces from a VAT number. Validators check it, for example a UK VAT check digit. You define them once on the Validation tab, then add the ones this field uses to its lists:

• Formatters (applied in this order)
• Validators (all must pass)

Each list shows only the chosen items. Click Add… to add one, use the arrows to move it up or down, and Remove to take it off. Formatters run from the top of the list.

See Validation rules.

Validation

The Validation section offers quick checks without defining a validator:

Dependent lists

A list can depend on another field. For example, Town offers only the towns of the Region chosen. Enter the other field's name in Limit the list by the value of field, then write each allowed value as value of that field | value:

North | Leeds
North | York
South | Bath
Other

A value without | (such as Other) is offered whatever the other field holds. While the other field is empty, the whole list is offered. The other field can be a field of the same document type or a batch field.

Default values with variables

A default value can hold parts that are filled in when the field gets its default: when the batch is created (batch fields) or when the document is first read (document fields). Values operators type are never changed.

For example, {date:dd/MM/yyyy} fills a received date with today's date, and BOX-{barcode} a box number from the barcode on the first page. An unknown part, such as {dat}, is refused when you save the field.

Tip: For dates on international documents, list every format you expect, for example dd/MM/yyyy, yyyy-MM-dd, dd.MM.yyyy and MMM d, yyyy.

Calculated values

With Advanced options on, Calculated value / cross-field check (script) takes a short JavaScript. Return a value to set the field, or throw a message to fail validation:

return Number(fields.NetAmount) + Number(fields.VatAmount);

For checks across several fields, validation rules are usually clearer.

Scripts on this field

Once saved, the field editor lists Scripts on this field: event scripts that run when the field gets or loses the cursor, changes or is checked. See Events and scripts.

Check your fields

Test extraction on real documents before you release: use the Finders tab or the zone designer for single fields, and the Test tab for the whole setup. See Testing a Batch Setup.