Enterprise Modernization Multi-Agent System
An enterprise-oriented agentic AI reference architecture for assessing legacy applications against engineering standards while keeping evidence, approvals, and consequential actions explicitly controlled.
View on GitHubProblem
Automated modernization assessment — using AI to review legacy applications against engineering standards — creates two major risks that most naive implementations ignore:
- AI-generated findings may not actually be grounded in source evidence.
- AI systems should not be allowed to autonomously perform consequential change-management actions.
This system was designed around those two risks from the ground up, rather than bolting on safety after the fact.
Architecture & Technical Highlights
engineering standardssynthetic sample apps
embeddingschunking
TypeScript UI
LangGraph run
execution traces
Orchestration
- LangGraph orchestration across the multi-agent workflow
- Parallel architecture and security analyst agents
- A modernization planner agent that synthesizes findings into a plan
- A bounded revision loop, capped rather than open-ended
Retrieval
- Code and engineering-standard ingestion into a structured knowledge base
- PostgreSQL + pgvector retrieval over ingested standards and application evidence
- Retrieval-augmented generation over the retrieved application and standards evidence
Evidence Grounding
- Evidence explicitly attached to every finding
- File path and line-level citations for traceability back to source
- A deterministic evidence reviewer that mechanically checks findings against retrieved chunks
Human Control
- Checkpointed workflow state with pause/resume for long-running assessments
- Explicit human approval gate before any consequential action
Integration
- FastAPI backend with a React / TypeScript frontend
- Mocked ServiceNow ticket creation for change-management follow-up
Engineering
- Docker for local/dev environments
- GitHub Actions for CI, including the evaluation and regression suite
- Audit-oriented tracing of agent runs
The diagram mirrors the graph implemented in the repository — the same nodes, the same fan-out/fan-in, the same conditional edges. All sample applications, findings, and evidence are synthetic; the ServiceNow step is a mocked adapter, not a live integration.
I intentionally separated model reasoning from evidence and approval enforcement. Agents can analyze and propose, while deterministic components validate evidence and control consequential actions.
Governance & Safety
- Every finding requires attached evidence — no evidence, no finding
- Citation IDs are mechanically checked against retrieved chunks, not just asserted by the model
- Unsupported claims are removed rather than surfaced to the user
- Approval rules are deterministic (rule-based), not left to LLM judgment
- Any consequential action requires explicit human approval
- Ticket creation is idempotent and fully mocked — no real ticketing system is touched
- Traces stored in PostgreSQL are sanitized before persistence
- LangSmith is used only for development/demo observability, not production monitoring
- Agent work is bounded by retrieval and revision limits to prevent runaway loops
Evaluation & LLMOps
- A golden evaluation dataset of representative assessment scenarios
- Offline, deterministic evaluation — not relying on live LLM judging alone
- CI regression checks that run on relevant changes
- Testing of risk classification against expected outcomes
- Testing of the approval policy logic
- Evidence coverage checks (are findings actually grounded?)
- Tool safety checks around the mocked ticketing action
- GitHub Actions running the evaluation and regression suite
- LangSmith used for development/debugging observability, not as a production SLA metric
Limitations & Disclosure
This is a synthetic portfolio use case. All applications, findings, CMDB context, and change tickets are synthetic. The ServiceNow integration is mocked. It is not customer delivery and does not access or modify real enterprise systems.
Additional demo-scope limitations from the repository:
- No production authentication/authorization — this is a demo-grade access model
- Change-management integration (ServiceNow) is mocked end-to-end
- Retrieval corpus is sample-scale, not a full enterprise codebase
- Assessment execution is synchronous in this reference implementation
Repository
Explore the full implementation, including the agent graph, evidence pipeline, and evaluation harness.
View CDW-UseCase on GitHub