Define a usable taxonomy
Choose categories tied to a downstream action, such as routing an invoice, identifying a purchase order, or separating policies from reports. Write boundaries between neighboring types. A document can contain several roles, so decide whether labels apply to whole files, pages, or individual document segments.
Exclude shortcut signals deliberately
Filenames, folder names, logos, and export headers can reveal labels without requiring content understanding. Decide which signals the deployed model will legitimately receive. If evaluation is meant to test document content, review whether convenient metadata or repeated templates make the test easier than the real workflow.
Represent mixed and unknown documents
Include bundles, cover letters, unsupported types, and low-quality scans if they occur in scope. Define abstention or review labels instead of forcing every example into the nearest category. Annotators should have examples showing how to handle a purchase order embedded inside a broader correspondence package.
Compare performance across sources
Report results by document family and source organization, not only the aggregate category score. Keep closely related templates together where independent evaluation requires it. Share the taxonomy, intended routing action, and available input signals with Fiund so a sourcing discussion can identify the documents needed to test useful behavior. For a difficult classification pilot, include a cover page followed by an invoice and a shipping notice inside one PDF. Ask whether the task should split the file, assign multiple labels, or route it for review. The expected answer should follow the declared unit of classification rather than an unstated convenience rule.
Review checklist
- Write boundaries between adjacent classes
- Choose file, page, or segment labels
- Define unknown and mixed-document handling
- Inspect template and metadata shortcuts