Application map / Document ingestion

How Document Ingestion Works

The reserve-study pipeline: upload, extraction, ops review, reconciliation, release, and the customer archive decision.

What it does

An authorized user selects a property, uploads its latest reserve-study PDF, and can leave the page. Gallopify durably tracks the document, sends it to the extraction service, gives an internal operator a real-source review and reconciliation workflow, applies the approved study to the property's planner, and asks the customer to decide whether projects absent from the new study should be archived.

V1 extracts projected expenditures only. The extraction engine is a black box whose quality can improve separately; v1 does not import funding schedules, balances, revenues, bank transactions, or compliance fields.

Customer:  Select property + PDF → Uploading → Queued → Processing
           → In review → Projects updated
           → Archive decision required (if projects are absent) → Complete

Ops:       Unclaimed review → claimed by one operator → correct extraction
           → resolve every ambiguous match and validation error
           → approve exact result revision + exact reconciliation proposal
           → one transactional release

Product invariants

1. A document never disappears. Once its row exists, every transition is durable and every non-terminal document is recoverable after a crash or deploy.
2. No unreviewed extraction changes portal data. Ops approval is mandatory in v1 — no confidence-based auto-release.
3. Project identity survives. A matched project keeps its id; linked transactions remain linked.
4. The approved study is authoritative. Its values replace matched projects' values even when a customer edited the project since the previous study; the conflict is shown to ops and retained in the audit record.
5. Absence is not deletion. Projects absent from the new study remain active until the customer submits the archive decision; archive is soft and reversible.
6. Release is atomic and idempotent. A retry cannot duplicate projects or partially apply a study.
7. Approval cannot go stale silently. It binds to the exact extraction revision, reconciliation proposal, and project versions ops reviewed.
8. Environment boundaries are physical. Dev and production use different databases, source buckets, extraction deployments, and credentials.
9. The source shown to ops is real. The actual PDF sits beside the editable result; no fabricated citation highlights (the v1 engine provides no trustworthy ones).
10. Status is honest. Internal stack traces and engine details never appear in customer copy; authorized intake is available by default, never represented by shipping a disabled uploader.

The state machine

The stored states are deliberately more precise than the customer labels:

The allowed transitions between these states are exhaustive and validated with row locking in one service transaction; returned, complete, abandoned, and cancelled never reopen.

Reconciliation: six exhaustive outcomes

Approved extracted rows are compared first with all active projects for the property (manually created or study-imported alike), then leftovers against archived projects so a reappearing component restores its identity instead of duplicating. Each row and each project participates in at most one match:

The matching algorithm

Matching is deliberately conservative — it proposes work for ops, it does not create identity from a weak guess. False negatives create ops work; false positives corrupt identity.

1. Normalize names and categories (Unicode NFKC, lowercase, "&" → "and", punctuation removed, whitespace collapsed; no stemming, numbers kept).
2. Exact lock: a unique pair matching on normalized name, canonical category, and displayed start fiscal year is locked.
3. Weighted fuzzy score for remaining active pairs: 0.60 × name token-set similarity (0–100, one pinned implementation — the thresholds depend on it) + category (15 if identical canonical) + year (15 same / 10 within 1 / 5 within 2) + amount (10 within 5% / 7 within 15% / 3 within 30%), capped at 100.
4. Auto-propose only when the pair is each side's mutual best, scores ≥ 85, and each side's margin over its runner-up is ≥ 10.
5. Archived candidates are scored the same way afterward but always require explicit operator acceptance.
6. Any remaining active pair scoring ≥ 60 makes its connected component ambiguous (release-blocking); rows with no such evidence are new, projects not held by an ambiguous component are archive_candidate.

Thresholds are implementation constants gated by a human-labelled matching corpus: 100% precision required for automatic proposals; recall reported but not a ship blocker.

Manual edits: three-way comparison

For a project previously touched by a release, the latest release snapshot provides the base: if the current portal value differs from that base, the field is a manual-edit conflict — ops sees base, current, and incoming, and must acknowledge before approval applies the incoming value. Projects with no prior snapshot get an ordinary two-way diff. The release ledger, not a general versioning subsystem, supplies the history.

Ownership boundary

Gallopify owns the source-object key and document context, the state machine, immutable extraction revisions, ops claims/reviews/approvals, reconciliation proposals, releases, archive decisions, project rows, and notifications. The extraction engine owns only transient compute and its disposable artifacts — it never writes portal data and does not own review or release state. Gallopify integrates solely through an idempotent submit/status facade; the engine's other endpoints are internals, not a contract.

Object custody (S3 invariants)

• Per-environment, versioning-enabled private buckets with per-environment credentials; no shared default bucket. Application principal, extraction principal, and bucket must belong to the same environment, and isolation probes must prove dev credentials cannot read production objects.
• Keys from durable ids only (documents/{property}/{document}/source.pdf); user filenames are display metadata and never appear in a key.
• Bytes never transit the backend: the backend creates the document row, then issues a 15-minute presigned POST constrained to the exact key, PDF content type, and a 1 byte–100 MB size range; the browser uploads directly to S3.
• Exact-version pinning: completion supplies the object version id; the backend verifies metadata and magic bytes, stores the source hash, and pins the version. Extraction, review, download, and release always use the pinned version — a later write to the same key cannot change reviewed input.
• Scoped access: Gallopify holds only presign-create, prefix-limited version enumeration, exact-version get/head, and exact-version delete; the engine holds read-only exact-version get for its environment prefix. Customer/ops downloads are authorized immediately before a five-minute exact-version presigned GET.
• Retention: source PDFs live for the active-property lifetime plus 90 days after archival or account deletion; immutable result/review/release/audit history follows the financial-record retention policy (at least seven years). Storage operations fail closed with typed errors.

Availability and the emergency pause

A capabilities endpoint is the sole frontend availability source: every authorized active property reports enabled — there is no allowlist, canary cohort, or rollout mode. The only override is a global emergency pause flag (UPLOAD_PIPELINE_V2_UPLOADS_PAUSED): pause rejects new presigns while history, in-flight processing, review, archive, restore, and cleanup continue. It is an incident brake, not a rollout mechanism — and it is the flag currently holding production intake closed (see the runbook).

Eligible historical uploads from the retired legacy pipeline are imported metadata-only as read-only "Earlier uploads" — property-unassigned, visible only to their original uploader, never able to poll, retry, release, or block a v2 upload. The importer never copies source objects or invents provenance.

Authority and other locked decisions worth knowing

• Any current canSee user may upload for a property — including assigned property and portfolio managers; upload does not require org-admin status. Property is selected and authorized at upload; release never infers it from the organization.
• Archive/restore authority starts with the uploader and uploading-organization admins, falling back to current HOA admins, then CMC admins.
• The v1 engine supplies no quarter: an extracted year becomes a full-fiscal-year project under the property's fiscal convention — quarter-level precision is never invented.
• Ordinary project deletion is blocked only while that project has a pending archive decision.
• Bank and budget ingestion remain unavailable until separately designed; /uploads is reserve-study-only.