Scriba, the officer-facing workspace, in one loop: a pending proposal is sent back, the agent searches the legislation and rewrites the passage live, the tracked edit is reviewed against the article that grounds it, approved, and published.
Highlights#
- Designed a human-gated AI editing system for Italy’s official procedure catalogs: an LLM agent drafts changes grounded in the legislation that justifies them, and only an officer-approved squash-merge can reach the record — the agent is structurally incapable of touching it.
- Reframed legal editing as code review: git is canonical, each editing task is a branch in its own worktree, the audit trail is commit history, and the review UI is a diff.
- Caged the agent’s tool surface to read/edit/grep plus one search tool — no shell, enforced by a runtime assertion that throws, with each session sparse-checked-out to a single environment and no parent-system credentials.
- Built an idempotent, watermark-based sync engine back to the parent CMS that halts on suspicious diffs instead of guessing.
The problem#
The Catalogo delle Procedure at procedimentipa.gov.it is the Italian government’s public catalog of procedimenti amministrativi — the official descriptions of how to obtain a permit, register a business, claim a benefit. Each procedure is anchored to specific articles of legislation, and the catalog is the record the administration answers for: citizens act on what it says. Behind the public site, the catalog is authored and maintained by domain experts on a dedicated platform; this system is built for them, not for citizens.
Today that content is edited directly in a CMS. There is no review trail, no isolation between editors, and when a law changes, the citations pointing at it rot silently. This is exactly the kind of content an LLM is good at improving — filling gaps, repairing citations, keeping prose consistent with the underlying legislation — and exactly the kind where an LLM failure is worst. A hallucinated legal citation in the official record is not a bug; it’s a liability.
So the brief was never “add an AI assistant to the CMS.” It was: put an AI assistant in front of the catalog while making it structurally impossible for the assistant to touch the live record on its own. Not prompted to be careful — incapable. Every design decision below is a consequence of that constraint.
The design#
The core move is a reframe: treat editing legal prose like reviewing code.
An officer opens a task and chats with the agent. Every agent turn that changes content surfaces as a diff. The officer accepts the turn (it becomes a commit on the task’s branch), rejects it with feedback (the changes are discarded and the feedback becomes the agent’s next prompt), or discards the task entirely. When the work is done, submitting squash-merges the branch into main. That reviewed merge — never the agent — is the only thing that can reach the official record.
One task, end to end: turns surface as diffs, a rejection’s feedback becomes the agent’s next prompt, a merge conflict becomes agent work, and the watermark only advances when every change lands.
Taking the reframe seriously is what pushed the content out of Postgres and into git. Once procedures live in a git repository, three problems that would otherwise need engineering come for free: session isolation is a worktree, the audit trail is commit history, and the review UI is a diff. Three invariants hold this together:
- Git is canonical. Postgres in the parent CMS is a derived read-model.
- One writer. The sync workflow is the only process that writes to Postgres.
- Session = task = branch. Each editing task gets its own worktree on its own branch, sparse-checked-out to a single environment.
The trust boundaries, drawn: the agent works inside a per-task cage; the only way into git main is an officer-approved squash-merge; the only writer to the parent CMS is the sync runner — and from the cage to the parent CMS there is no path at all.
The price of git-as-canonical is owning the one-way sync back to Postgres, because the parent CMS still serves the public read path. I treated it as replication, not scripting: each environment has a watermark that advances only when every change in a window lands, every write is validated first, and when the diff looks wrong the sync halts and waits for an operator instead of guessing — a window containing only deletes is treated as a red flag, not an instruction. I built the receiving side too, adding the atomic bulk-PATCH and validate endpoints to the parent’s Python/FastAPI service, so the contract is honest at both ends.
The agent itself is deliberately caged, and moving to a filesystem is what made the cage tractable. Its tool surface is read, edit, write, grep, find, and one search tool over a per-environment full-text index. No shell, ever — enforced by a runtime assertion that throws if a dependency update ever re-enables one, not just by configuration. Sparse-checkout closes the worktree over a single environment at session start, so a staging session cannot reach prod. Agent processes never hold parent-system credentials. And search results from the vault are wrapped as untrusted data before the agent sees them, a hedge against instructions smuggled into legislation text.
Deliberate non-goals. Not a real-time collaborative editor: one officer, one task, one branch, with concurrent edits reconciled at submit time. Not a general-purpose coding agent: widening the tool surface — a shell, cross-environment search, parent-API access from the agent — is refused in code, not just in docs.
What was hard#
Three things, all variants of the same underlying problem: keeping durable state coherent around a non-deterministic, long-running agent.
Merge conflicts. A submit squash-merges the session branch into main, and main may have moved since the branch was cut. The rule was set upfront: a conflict must become agent work, never an auto-merge. On conflict the pipeline aborts the merge, preserves the worktree and branch, releases the submit lock, and re-prompts the agent with a plain-language description of the conflicting files — turning a stranded submit into one more reviewable turn. An integration test pins the whole contract against real git, down to “the agent was re-prompted exactly once.”
The sync engine. A long-lived process writing to someone else’s API will crash mid-run, hit expired tokens, and get retried. The engine is idempotent by construction: every run recomputes its diff from the stored watermark, so retries and crashes are safe. Transient failures retry with jittered backoff and honor Retry-After. A 401 gets exactly one forced token refresh; a second 401 halts rather than loops. Bursts of submits coalesce per environment into at most one in-flight run plus one pending follow-up.
Dying well. An LLM stream has no natural end, so a graceful shutdown can hang forever on a stuck stream. On SIGTERM the server races each streaming session’s abort against a 10-second timeout. Sessions mid-submit are deliberately left alone — a durable pipeline is safer reconciled at next boot than torn down mid-merge. Boot recovery flips orphaned run rows and enqueues catch-up syncs for any environment whose watermark trails git HEAD. The test for this spawns a real subprocess with a stream rigged to hang forever, sends SIGTERM, asserts a clean exit within 15 seconds, restarts, and checks that the task and its worktree survived.
Status and evidence#
In active development, pre-alpha. The backend loop works end-to-end through the operator console: create a task, watch tokens and tool calls stream over SSE, review the git diff, accept, submit, sync.
- The HTTP surface is typed end-to-end with zod-openapi and serves its own interactive API reference.
- The content vault bootstraps from the live catalog behind it — around 700 procedures today.
- The sync path has a rollback runbook with a 30-minute recovery target.
- Scriba, the officer-facing UI, is a working prototype with the backend wiring specced. It borrows the idiom of an AI code editor: the agent’s work — legislation searches, articles read, quoted excerpts — is inspectable before the rewrite is even proposed, and tracked edits land inline in the document with approve / request-changes / publish as first-class actions.
- Scriba streams over the backend’s own SSE event vocabulary, so the prototype’s mock emitter swaps directly for the real backend; it ships in light and dark themes with an IT/EN switch.
Screenshots#
Stack#
TypeScript, Bun, Hono, zod-openapi, Scalar, bun:sqlite, SSE, simple-git, pi-coding-agent (the sandboxed agent runtime sessions run on), React 19, Python/FastAPI (parent-side endpoints).