v1.0 — Architecture & delivery plan (ACCEPTED)¶

Supersedes v0.1-discovery. Data model and corpus: v0.2. Change history: CHANGELOG. Decisions with lasting consequences: docs/adr/.

1. What we are building¶

LedgerLens MCP — a production-grade Model Context Protocol server for Kosha Financial, a fictional Indian neobank. AI agents use it to:

look up customers, accounts, KYC state and transactions (synthetic, regulator-realistic data);
investigate AML alerts and screen entities against sanctions/PEP lists;
raise disputes and flag transactions (idempotent, audited writes);
search ~85 internal SOPs, policies and API docs across 9 departments (hybrid RAG with citations).

It exists to demonstrate production engineering: OAuth 2.1 auth, scope-level authorization, PII masking, resilience patterns, observability, IaC, CI/CD and a complete test pyramid.

2. Decision summary¶

Area	Decision
Language / tooling	Python 3.13, `uv` workspace monorepo, `poethepoet` tasks, `ruff`, `pyright` (strict), `pytest` + `hypothesis`
MCP server	FastMCP, Streamable HTTP, stateless JSON-response mode
Domain API	FastAPI "Core Banking API" — the MCP gateway calls it over HTTP
Third-party sim	`screening-sim` — FastAPI sanctions/PEP vendor with tunable latency & failure
Compute	AWS Lambda container images (arm64) + Lambda Web Adapter; same image runs in `docker compose`
Edge	CloudFront → API Gateway HTTP API → Lambda. No ALB (API Gateway is the managed front door)
Auth	MCP Authorization spec: server = OAuth 2.1 Resource Server, RFC 9728 metadata; Cognito via FastMCP's Cognito/OAuth-proxy provider; scopes + roles; JWT validated in-app
System of record	RDS PostgreSQL 16 `db.t4g.micro` single-AZ + pgvector
Hot control state	DynamoDB: per-client token-bucket rate limits, breaker force-open overrides
Embeddings	OpenAI `text-embedding-3-large` @ 1024 dims, HNSW; key in Secrets Manager
Retrieval	Hybrid pgvector + Postgres FTS fused by RRF in one SQL query; RLS on document classification; degrades to FTS-only
Async	SQS + DLQ: (1) document ingestion, (2) transactional-outbox domain events. No Kafka
Network	VPC, private subnets for Lambda + RDS, NAT instance (t4g.nano), free S3/DynamoDB gateway endpoints
IaC	AWS CDK (Python) + `cdk-nag`
Deploy	GitHub Actions → OIDC → `cdk deploy`; CodeDeploy canary (10% for 5 min) on Lambda aliases with alarm-based automatic rollback
Observability (AWS)	Powertools for AWS Lambda (logs, metrics, idempotency) + OpenTelemetry → X-Ray; CloudWatch dashboard & alarms
Observability (local)	OpenTelemetry Collector → Prometheus + Grafana + Tempo, provisioned dashboards (compose profile `observability`)
Agent observability	Langfuse (LangGraph agent traces)
Clients	LangGraph "Kosha Ops Copilot" (primary), Google ADK example, Streamlit console UI, Claude Desktop config
Load testing	Locust (Python) — replaces k6 to keep one language
Docs	MkDocs Material on GitHub Pages; Mermaid + Excalidraw + animated SVG infographics
Region / account	Single AWS project, Region `eu-north-1`, FREE plan
Budget	≈ $25/month while running; `poe pause` / `poe resume` stop RDS + NAT when not demoing

Tooling questions answered¶

Tool	Verdict	Reason
ngrok	Dev-only, optional (`poe tunnel`)	Lets claude.ai reach a local server during development. Never part of production; the deployed CloudFront URL replaces it
Prometheus + Grafana	Yes — locally	Vendor-neutral OTel instrumentation feeds Grafana dashboards in compose (great screenshots). In AWS, Amazon Managed Grafana is not available on the plan and self-hosting costs money, so the same OTel data goes to CloudWatch/X-Ray
Load balancer (ALB)	No	~$18/month for nothing API Gateway + CloudFront don't already provide for Lambda
CodeDeploy	Yes	Canary traffic shifting + automatic rollback on CloudWatch alarms — real production deployment safety at near-zero cost
CodePipeline	No	GitHub Actions is the pipeline; adding a second orchestrator adds nothing
Kafka / MSK	No	Paid-plan only, ~$100s/month, no streaming workload

3. Deployables¶

Deployable	Kind	Responsibility
`mcp-gateway`	FastMCP on Lambda	MCP protocol, OAuth resource server, scope checks, masking, audit, rate limiting, resilience toward downstreams
`core-api`	FastAPI on Lambda	Domain logic & persistence: customers, accounts, ledger, KYC, AML, disputes, KB retrieval, outbox
`screening-sim`	FastAPI on Lambda	Simulated external sanctions/PEP vendor with chaos knobs
`ingest-worker`	Lambda (SQS)	Parse → chunk → embed → upsert documents
`outbox-relay`	Lambda (scheduled)	Publish `ops.outbox` rows to SQS exactly-once-effect
`events-worker`	Lambda (SQS)	Dispute SLA timers, review queue fan-out

4. MCP surface¶

Tools — Customer & KYC: search_customers, get_customer_profile, get_kyc_status, list_kyc_documents · Accounts & payments: get_account_summary, search_transactions, get_transaction, get_spending_insights · Risk: list_aml_alerts, get_aml_alert, screen_entity · Writes: raise_dispute, flag_transaction_for_review · Knowledge: search_knowledge_base, get_document · Ops: get_service_health Resources — kb://{department}/{doc_code}, customer://{customer_id} Prompts — investigate_suspicious_activity, kyc_periodic_review, dispute_triage

Scopes and roles¶

Scope	support_agent	analyst	compliance_officer	demo_readonly
`customers:read`	✓	✓	✓	✓
`transactions:read`	✓	✓	✓	✓
`kyc:read`	✓	✓	✓	✓
`kyc:pii` (unmasked PAN/contact)			✓
`aml:read`		✓	✓
`screening:run`		✓	✓
`disputes:write`	✓		✓
`reviews:write`		✓	✓
`kb:read` (classification ceiling)	internal	confidential	restricted	public

5. Resilience design¶

Pattern	Where	Notes
Deadline propagation	every hop	Request budget set at gateway; each downstream call gets `min(remaining − margin, per-call cap)`
Timeouts	httpx, psycopg, OpenAI	Connect / read / pool timeouts set explicitly; never library defaults
Retry	idempotent calls only	Exponential backoff + full jitter, max attempts, retry budget (≤ 10% extra load)
Circuit breaker	per downstream, per instance	closed → open → half-open; failure-rate window; DynamoDB force-open override shared across instances
Bulkhead	per downstream	`asyncio.Semaphore` caps; Lambda reserved concurrency caps DB connections
Rate limiting	per OAuth client	DynamoDB token bucket (60 req/min, burst 20) → `429` + `Retry-After`
Idempotency	write tools	Key + request hash stored in the same Postgres transaction as the write
Load shedding	gateway	Reject early when in-flight > limit or remaining budget too small
Graceful degradation	KB search, screening	FTS-only search when OpenAI fails; `screen_entity` returns `status: deferred` when vendor breaker open
Fault injection	gateway + sim	Signed chaos header / config: latency, errors, timeouts per dependency

6. SLOs¶

Indicator	Target
p95 latency, warm, simple read tools	< 300 ms
p95 latency, `search_knowledge_base`	< 900 ms
Cold start	< 2.5 s
Availability (error budget)	99.5%
Rate limit per client	60 req/min, burst 20

7. Repository layout¶

ledgerlens-mcp/
├── packages/                 # importable libraries (no I/O at import time)
│   ├── kosha-domain/         # money, ids, statuses & state machines, validators, masking
│   ├── kosha-resilience/     # deadline, retry, breaker, bulkhead, rate-limit interfaces
│   ├── kosha-observability/  # logging, tracing, metrics setup shared by services
│   ├── kosha-datagen/        # deterministic synthetic data generator
│   └── kosha-kb/             # parsing, chunking, embedding, hybrid retrieval
├── services/
│   ├── core-api/             # FastAPI + Alembic migrations
│   ├── mcp-gateway/          # FastMCP
│   ├── screening-sim/        # FastAPI vendor simulator
│   └── workers/              # ingest-worker, outbox-relay, events-worker
├── agents/                   # LangGraph copilot, ADK example, Streamlit console
├── infra/                    # AWS CDK app (Python)
├── corpus/                   # SOP / policy / API source documents
├── tests/                    # cross-service e2e, contract, load (Locust), chaos
├── docker/                   # shared Dockerfile, compose overrides, observability config
└── docs/                     # MkDocs site: HLD, LLD, ADRs, plans, diagrams

8. Test strategy¶

Layer	Scope	Tooling
Unit	domain rules, validators, masking, state machines, resilience primitives	pytest, hypothesis (property-based)
Integration	repositories, migrations, SQL retrieval, outbox, SQS consumers	testcontainers (Postgres+pgvector), LocalStack
Tool	every MCP tool via FastMCP in-memory client with real core-api	pytest-asyncio
Contract	tool/resource/prompt schemas snapshot; OpenAPI snapshot	syrupy
E2E	deployed stack, real Cognito tokens, every tool	pytest + MCP client
Load / chaos	SLO verification; breaker open → half-open → close	Locust, chaos header
RAG eval	recall@k, MRR, faithfulness on golden set	pytest + DeepEval

Per-tool failure matrix (each row tested for each tool where applicable): missing / expired / wrong-audience token · missing scope · masking per role · invalid input · not found · empty result · oversized result truncation · tampered cursor · downstream timeout · downstream 5xx · breaker open · rate limited · idempotent replay · idempotency conflict · concurrent duplicate · OpenAI outage → degraded · DB pool exhausted · prompt-injection content in KB results.

Coverage gate: 90% lines on packages/ and services/.

9. Delivery phases¶

Phase	Delivers	Exit criteria
P0	Repo, uv workspace, CI/CD, Docker build, HLD/LLD, ADRs, glossary, docs site shell, CLAUDE.md	CI green; images build; docs build
P1	Generator, schema + Alembic migrations, compose stack	`poe seed` loads full dataset; ledger reconciles
P2	Core API, test-first	All routes + failure tests green
P3	MCP gateway: tools/resources/prompts, auth, masking, audit	Tool matrix green against compose
P4	Resilience package, chaos, screening-sim	Breaker lifecycle demonstrated in tests
P5	Corpus, ingestion pipeline, hybrid retrieval, RAG evals	Eval thresholds met in CI
P6	CDK, OIDC deploy, CodeDeploy canary, e2e, Locust	Deployed; SLOs measured
P7	LangGraph + ADK agents, Streamlit console, demo GIF	Demo script runs end to end
P8	README hero, infographics, animations, docs polish	Recruiter-ready