Samples and classifier training

DocView Capture classifies documents with rules by default: keywords, patterns, barcodes and anchor text (see Document classes and classification). Rules are easy to understand and work well when every document type has a clear title or barcode.

Some document sets are harder. Letters, contracts, medical reports or correspondence often have no fixed title, and the same words appear on many types. For these, you can train a text classifier from sample documents. DocView learns which words and phrases tell your document types apart, so you do not have to write rules for them.

The classifier is trained with machine learning inside your DocView installation, from the OCR text of your samples, read with the Batch Setup's OCR settings. The same samples are used to test finders.

Where to find it

Open the Batch Setup, go to the Document classes & fields tab and choose Samples and training at the top. (Classes and fields next to it is where you set up the types themselves.) Three steps run across the top:

1. Samples — your document types one under the other, each with its number of samples (red: too few to train, amber: more help, green: ready at 10 or more). Click a type to see its samples.
2. Train — what the model will learn from, Train a new model, the models kept so far and the report of the one you choose.
3. Test — documents the model has not seen, classified side by side by this version's set-up, the rules and the models.

Create a draft first if the version is released: samples and models can be looked at in any version, but only a draft can be changed.

Add samples

To one document type: click the type in the list, then drop files on the area at the top of its samples (or anywhere on the page, or on the type's name in the list), or click Add files. They are added at once, with a progress bar; each file is read with OCR as it arrives, so allow a few seconds per file. PDF, TIFF, JPEG, PNG or a ZIP of them, up to 25 MB each; the first 20 pages of a file are read.

Many types at once: drop a folder of folders, one per document type and named like the types, or click Add a folder of types:

Samples\
  Invoices\      invoice-001.pdf, invoice-002.pdf …
  Letters\       letter-a.tif, letter-b.jpg …
  Credit notes\  …

DocView lists each folder with its number of files and the type it goes to — Invoices finds the type Invoice; capitals, spaces and plurals do not matter. For a folder with no matching type, choose a type, New document type… or Skip, then click Add N samples. A ZIP of the folder works the same way. New document type under the list adds a type without a folder.

Files that are not PDF, TIFF, JPEG or PNG, or larger than 25 MB, are left out and listed.

Look at a sample

Click a sample to open it: the page on the left, the text read from it (OCR) on the right, and what the last training and the last test made of it. Use the arrow keys or ‹ › to go through the samples of the type. Then:

• Use for training — switch off to leave a sample out of training (unreadable, or not typical); it stays for finders and tests and is shown faded with Left out. The filters All / Used / Left out above the samples show either kind.
• Move — file it under another type when it is in the wrong one.
• Delete sample — removes it from training, finders and tests.

A sample with very little text will not help the classifier; the viewer warns when it has fewer than 50 characters.

Samples whose type is not in this version are listed under Samples of other types; open them to move or delete them.

Tip: Samples uploaded on the Finders page belong to the same set. Test finders on these samples in a type's … menu runs its finders on them.

Train a model

On 2 Train, What it learns from lists every type with the samples used (samples left out are not counted).

1. Optionally type a note, for example more letters from new suppliers.
2. Click Train a new model.

Training needs at least two document types with 2 or more samples each; types with fewer are left out. DocView trains, tests the model with cross-validation, and keeps it as the next numbered model with its report. The draft then uses the new model. Earlier models stay in the list, so nothing is lost by training again.

The training report

Cross-validation means DocView holds back part of the samples, trains on the rest, and tests on the part it held back, several times over, so every sample is classified by a model that never saw it. Then it trains the final model on all samples.

Choose the model a version uses

Each model shows In use when this version uses it and used by v2, v3 for other versions. Click Use on another model to switch the draft to it, or Use on Rules only (no model) to classify with the rules. Released versions keep the model they were released with. A model no version uses can be deleted.

How the model and the rules work together is set on the OCR step of the workflow, under Classification: rules only, the model only, or one first and the other when the first is not sure. Automatic uses the model whenever the version has one.

Compare two models

On 2 Train, open Compare two models, choose an earlier and a later model and click Compare. You see:

• the accuracy of each, and the change in points;
• per type, the samples and the recall and precision of both, better in green and worse in red;
• Now right: samples the later model gets right that the earlier one got wrong; Now wrong: the other way round; and how the samples added since were classified.

Add samples, train again, and compare: keep the better model in use.

Test with new documents

On 3 Test, drop documents that are not samples (or choose files, or a folder). Each is read and classified by this version's set-up, the rules only and the newest models, shown side by side with their confidence; nothing is stored. To see right and wrong, drop a folder with a sub-folder per document type (named like the types), or choose the Expected type for files without one: right answers turn green, wrong ones red, and a last row shows how many each got right. Up to 20 documents of 10 MB at a time; the first 5 pages of each are read.

Check the samples themselves below classifies every sample with the set-up, the rules or one model. A model knows the samples it was trained on, so this mostly checks the rules, and samples added after the last training.

Improve a weak model

• Add more samples to types with low recall or precision, from as many senders and layouts as possible.
• Look at the samples it got wrong: move misfiled ones, delete unreadable ones.
• Merge types that are truly the same, or split one type that covers very different documents.
• Train again and compare with the previous model.

Models, versions and copies

• A new draft keeps the model of the version it was made from.
• Samples belong to a type by its Key, across versions. Keep keys stable so your samples stay with the right type.
• Batch Setup export files do not include the models or the samples. After you import or clone a Batch Setup, add samples and train again on the new copy. See Import and export Batch Setups.

Rules or a trained model?

Many setups start with rules and add a model later, once enough real documents have been collected.

The Classification change separation rule also uses classification, so a good model can improve document separation of unmarked stacks too. See Image cleanup and separation.