Microservices, shared libraries, infrastructure-as-code, frontend apps, data pipelines, all in separate git repos. Start Claude Code in one and ask about another, and it has no context. It doesn’t know the workspace exists.

Here’s how I’ve been setting this up to work across repositories.

The problem

When you start Claude Code in orders/order-service, it has no idea that orders/orders-ui exists next door, or that shared libraries live in shared/, or that the data team’s Spark jobs are in analytics/. Every session starts with you explaining the workspace layout.

The same problem shows up when someone new joins the team. They clone one repo, but they don’t know what other repos exist, how they relate, or where to look for shared infrastructure.

A bootstrap repo as the workspace root

The approach I landed on: a bootstrap repo that sits above all the other repos as the workspace root. It doesn’t contain application code. It contains:

1. A repo manifest listing every repo, where it lives, and what it does
2. Context files that Claude Code picks up from the directory tree
3. Tasks for common cross-repo operations (pull all, search all, check status)

I use mani as the repo manager, but the ideas apply regardless of tooling. You could do this with a shell script and a list of repos.

Directory structure

workspace/
  mani.yaml                  # imports per-product configs
  CLAUDE.md                  # org-level context
  mani.d/
    orders.yaml              # order management (3-tier)
    shipping.yaml            # shipping & logistics (3-tier)
    analytics.yaml           # data platform (Spark, Airflow, APIs)
    assist.yaml              # agentic AI system (FastAPI, LangGraph, React)
    shared.yaml              # shared libraries and services
    infra.yaml               # infrastructure repos
  orders/
    CLAUDE.md                # team-level context (tracked in bootstrap)
    order-service/           # Spring Boot (gitignored)
    payment-service/         # Spring Boot (gitignored)
    orders-ui/               # React (gitignored)
    reporting-service/       # Spring Boot + PostgreSQL (gitignored)
    pricing-engine/          # Vert.x, not Spring Boot (gitignored)
  shipping/
    CLAUDE.md
    shipment-service/        # Spring Boot + MongoDB
    shipping-ui/             # Angular
    carrier-service/         # Spring Boot, reactive
  analytics/
    CLAUDE.md
    airflow-dags/            # Python, Airflow
    spark-jobs/              # PySpark on EMR
    metrics-service/         # Kotlin, Micronaut
    dashboard-ui/            # React
  assist/
    CLAUDE.md
    agent-service/           # FastAPI + LangGraph
    conversation-service/    # Spring Boot + WebSocket
    chat-ui/                 # React + streaming chat
  shared/
    CLAUDE.md
    react-lib/
    java-commons/
    feature-toggles/
  infra/
    CLAUDE.md
    terraform-modules/
    ci-templates/
    cluster/

Each indented directory under a product (order-service/, orders-ui/, spark-jobs/, etc.) is a separate git repo, cloned by the repo manager and gitignored by the bootstrap repo. The CLAUDE.md files at each level are tracked in the bootstrap repo.

Three layers of context

Claude Code walks up the directory tree looking for CLAUDE.md files. If you start it in orders/order-service, it reads:

1. orders/order-service/CLAUDE.md (repo-level, committed in that repo)
2. orders/CLAUDE.md (team-level, committed in bootstrap)
3. workspace/CLAUDE.md (org-level, committed in bootstrap)

Each layer adds context without repeating what the others provide.

Layer 1: Organisation

The org-level CLAUDE.md covers things that apply everywhere:

• Warning that this is a multi-repo workspace (check git rev-parse --show-toplevel before git operations)
• How to discover repos (point to the manifest file)
• Which products exist and what they own
• Common cross-repo operations

Keep this short. Claude reads it on every session regardless of which repo you’re in.

Layer 2: Team

The team-level CLAUDE.md covers conventions shared across repos in that group. The content varies by product type:

A 3-tier product (like orders or shipping) might cover:

• Backend stack (Java 21, Spring Boot 3.5, Gradle, MongoDB)
• Frontend stack (React 19, Vite, TypeScript)
• Build and test commands for each
• The one exception (the pricing engine uses Vert.x, not Spring Boot)

A data platform (like analytics) might cover:

• Orchestration (Airflow DAGs, triggered via async-job-service)
• Processing (PySpark on EMR, containerised Python jobs on ECS)
• Multi-region support (pipelines run per-region with region-specific config)

An agentic system (like assist) might cover:

• Agent framework (FastAPI + LangGraph for orchestration)
• Backing services (Spring Boot for persistence, WebSocket for streaming)
• Frontend (React with streaming UI patterns)

I learned not to list repos here. Lists go stale. Instead, tell Claude where to look: “This group’s repos are defined in mani.d/orders.yaml. Each project has a desc field. Check that file for the current list.”

Layer 3: Repository

This lives in each repo and is maintained by the team that owns it. Build commands, architecture notes, test instructions, things specific to that codebase. This is standard Claude Code usage, nothing new.

Project descriptions in the manifest

One-line descriptions in the repo manifest make a big difference for discovery. When Claude reads the manifest, it knows what each repo does without cloning or exploring it.

projects:
  order-service:
    desc: Order lifecycle management and fulfilment
    url: git@gitlab.com:acme/order-service.git
    path: orders/order-service
    tags: [orders, java]

  pricing-engine:
    desc: Vert.x real-time pricing engine
    url: git@gitlab.com:acme/pricing-engine.git
    path: orders/pricing-engine
    tags: [orders, java]

  orders-ui:
    desc: React UI for order management and reporting
    url: git@gitlab.com:acme/orders-ui.git
    path: orders/orders-ui
    tags: [orders, ui]

The desc field costs almost nothing to maintain and saves Claude from guessing or asking.

Cross-repo tasks

A repo manager like mani lets you define tasks that run across repos:

tasks:
  update-repos:
    desc: pull latest for all repos
    target: all
    cmd: |
      current=$(git rev-parse --abbrev-ref HEAD)
      if [[ -n $(git status -s) ]]; then
        git fetch origin $branch
        echo "FETCHED (dirty working tree on $current)"
      elif [[ "$$current" != "$branch" ]]; then
        git fetch origin $branch
        echo "FETCHED (on branch $current, not $branch)"
      else
        git pull --rebase origin $branch
      fi

This one pulls latest on repos that are clean and on the default branch, and fetches (but doesn’t touch) repos with work in progress. The data is available locally either way, so the next pull is fast.

Other useful tasks: search across all repos, check which repos have uncommitted changes, trigger CI pipelines.

The gitignore trick for team-level CLAUDE.md files

The bootstrap repo gitignores all sub-repo directories. But the team-level CLAUDE.md files need to be tracked in bootstrap, inside those same directories. The fix:

# Use dir/* instead of dir/ so exceptions work
orders/*
!orders/CLAUDE.md

orders/ ignores the directory entirely (git won’t look inside). orders/* ignores everything inside it but lets you exclude specific files.

Skills, hooks, and commands

Claude Code supports skills, hooks, and custom commands configured in the .claude/ directory of a repo. These have always worked at the repo level. The bootstrap structure gives you two more levels:

Org level (in the bootstrap repo’s .claude/):

• Skills that work across all repos. I have one that queries SonarQube for any repo in the workspace, auto-detecting the project key from the current directory.
• Pre-commit hooks (gitleaks for secret detection, applied to the bootstrap repo itself).
• Shell scripts for operations that span teams, like auditing which repos still need a branch migration.

Team level (in each team’s CLAUDE.md or tracked config):

• Build conventions that apply to all repos in a team but not the whole org. A team with ten Spring Boot services can document the shared Gradle convention plugins once, in the team CLAUDE.md.

Repo level (in each repo, as before):

• Repo-specific skills, hooks, and commands. Nothing changes here.

The layering means you write a SonarQube skill once at the org level and it works in any repo. You document ./gradlew spotlessApply once at the team level and every repo in that team inherits the context.

Partial and full checkouts

Not everyone needs the whole workspace. Most developers I work with only clone their team’s repos:

workspace/
  mani.yaml
  CLAUDE.md
  orders/
    CLAUDE.md
    order-service/
    payment-service/
    orders-ui/

They still get the org-level and team-level CLAUDE.md files. Claude Code still understands the team’s conventions and knows how to discover the rest of the organisation through the manifest.

A platform engineer or architect who works across teams clones everything. They get the full context at every level.

The repo manager handles both. You can tag repos by team and clone selectively (mani sync --tags orders) or clone everything (mani sync). Either way, the layered context works because CLAUDE.md files at each level are already in place.

What this gets you

When someone starts Claude Code in any repo in the workspace, it already knows:

• What the repo does and how to build it
• What other repos exist in the same team and how they relate
• How to navigate to shared libraries, infrastructure, and deployment configs
• Common conventions and exceptions

If you want to try this, start small. Create a bootstrap repo, add a CLAUDE.md with your workspace layout, and list your repos in a manifest with one-line descriptions. You can add team-level context and cross-repo tasks as the structure proves useful.

That’s the gap. Teams building production AI workflows have developed patterns for making AI reliable. Developers using AI coding assistants like Claude Code, Cursor, or Copilot mostly haven’t adopted them yet.

These patterns aren’t theoretical. They’re practical and don’t require special tooling.

The Patterns

Here’s what each one looks like. Some link to deeper posts.

Deterministic Tool Delegation

The pattern: Don’t let the AI make decisions it doesn’t need to make. If a deterministic tool can handle something (refactoring, formatting, linting, data validation), use the tool. The AI’s job is orchestration, not execution.

What most developers do instead: Ask the AI to rewrite code for a rename, follow a style guide from memory, or process data it doesn’t need to see.

Why it matters: Every unnecessary decision is a degree of freedom. Every degree of freedom is an opportunity to get something wrong, burn tokens, and produce a result you can’t reproduce. Deterministic tools give you the same output every time.

I wrote about this in depth in The Unix Philosophy for Agentic Coding.

Verification Loops

The pattern: Instead of accepting the first output, create a generate-evaluate-revise cycle. The agent produces work, a separate pass critiques it against explicit criteria, and the agent revises.

What most developers do instead: Prompt, receive, accept or reject. The interaction model is single-shot.

Why it matters: LLMs produce plausible output that can be subtly wrong. Research shows 10-20 percentage point improvements on coding benchmarks from reflection alone. Anthropic’s own guidance identifies the evaluator-optimizer workflow as one of the core composable patterns.

What this looks like in practice: After asking your AI assistant to implement a feature, follow up with: “Review what you just wrote. Check for edge cases, error handling, and whether it follows patterns in this codebase. List problems, then fix them.” For high-stakes changes, use a separate session as an independent reviewer.

This pattern is also the foundation of test-driven development with AI: write the test first, let the AI implement, then the test itself becomes the verification loop. I’ve touched on this in the TDD workflow in intelligent Engineering: In Practice.

Context Engineering

The pattern: Deliberately architect what information the model sees, when it sees it, and in what form. Treat context as a finite resource, not an infinite scratchpad.

What most developers do instead: Paste entire files, full error logs, and broad descriptions, trusting the model to extract what’s relevant.

Why it matters: Including irrelevant data actively worsens output quality. Models have attention patterns that favour the start and end of context, with the middle getting less focus. More context is not always better context.

I wrote a full post on this: Context Engineering for AI-Assisted Development. The short version: curate your CLAUDE.md for signal density, use .claudeignore to exclude noise, provide the two or three most relevant files rather than the entire directory, and start fresh sessions when context degrades.

Upfront Planning

The pattern: Before any code is written, create an explicit plan that decomposes the work into steps with dependencies and acceptance criteria. Review the plan before execution begins.

What most developers do instead: Give the AI a single prompt describing what they want and let it figure out the approach. “Add user authentication” becomes one big prompt rather than a sequence of reviewable steps.

Why it matters: Internal planning by the model is invisible and unreviewable. An explicit plan is where you catch architectural mistakes that are expensive to fix after implementation. It also prevents the “AI rewrote half the codebase and something is broken but I don’t know where” problem.

What this looks like in practice: For any task that touches more than two files: “Before implementing, create a plan. List the files you’ll modify, the changes in each, the order of changes, and how you’ll verify each step works.” Review the plan before saying “proceed.”

This is central to the design discussion workflow I use.

Persistent Memory

The pattern: Retain lessons, decisions, and discovered patterns across sessions. Build institutional knowledge over time rather than starting from zero each conversation.

What most developers do instead: Every session starts fresh. They rediscover the same issues, re-explain the same conventions, and re-learn the same codebase quirks.

Why it matters: Without cross-session memory, the AI makes the same mistakes repeatedly and you correct it repeatedly. Codified constraints prevent the same mistakes from recurring.

What this looks like in practice: Maintain a CLAUDE.md that evolves. When you discover a gotcha (“the payments service returns 200 even on failures, check the response body”), add it immediately. When the AI makes a mistake, codify the prevention rule. Over time, your context docs accumulate the institutional knowledge that makes the AI genuinely useful on your specific project.

I cover this in detail in the Foundation and Context Documentation layers of the intelligent Engineering stack.

Structured Guardrails

The pattern: Define explicit boundaries around which decisions the AI can make autonomously and which it should escalate. This includes architectural constraints (“don’t introduce a new database without discussing it”), scope boundaries (“only modify files in this module”), and approval gates for high-impact changes.

What most developers do instead: Give the AI full autonomy without defining what’s in and out of scope. The agent makes architectural decisions, introduces new patterns, or changes public APIs without checking whether that’s what you intended.

Why it matters: A prompt might be ignored as context fills up. A pre-commit hook won’t be. Deterministic enforcement catches what prompt-based instructions miss.

What this looks like in practice: Define boundaries in your CLAUDE.md (“never modify migration files without asking”). Use pre-commit hooks for formatting, linting, and security checks. Set up Claude Code hooks for auto-formatting and blocking sensitive operations. Let low-risk operations run freely. Pause high-risk ones for review.

I wrote a hands-on tutorial on this: Level Up Code Quality with an AI Assistant.

Observability

The pattern: Systematic tracking of what the AI did, what worked, what failed, and using that data to improve future interactions.

What most developers do instead: Look at the output. No structured feedback, no trend tracking, no quality measurement over time.

Why it matters: The METR study found developers estimated they were 24% faster with AI when they were actually 19% slower. Gut feel is unreliable. Without measurement, you don’t know if the AI is helping, and you can’t systematically improve your workflows.

This is the least mature pattern in the list. The tooling barely exists for individuals and is fragmented across teams. I explore the current state, the gaps, and what we’d like to see in Observability for AI-Assisted Development.

Multi-Agent Specialisation

The pattern: Instead of one generalist agent handling everything, use multiple specialised agents with focused context, specific tool access, and defined roles.

What most developers do instead: One session, one agent, planning, implementation, and review all in the same context window.

Why it matters: Each agent gets a fresh, focused context window rather than one bloated context trying to hold planning, implementation, review, and testing simultaneously. Specialisation also lets you use different models for different tasks (a thinking model for planning, a fast model for implementation).

What this looks like in practice: Claude Code recently started offering to clear context when you accept a plan, giving the implementation phase a fresh, focused window with only the plan carried forward. Planning and implementation benefit from separate contexts.

Take it further. Build an agentic team with a backlog: a planning agent that decomposes work into tasks, implementation agents that execute them, QA agents that test, and review agents that validate. Each agent has specific skills and focused context for its role. Claude Code’s Agent Teams and subagent features support this natively. Anthropic’s engineering team built an entire C compiler using 16 agent teams, producing 100,000 lines of Rust code. Codex has similar multi-agent capabilities.

Anthropic’s internal benchmarks showed a 90% improvement with multi-agent (Opus lead + Sonnet subagents) over solo Opus on complex tasks. Tekion deployed persona-driven agents across 1,300 engineers and saw 50-85% productivity gains, compared to 30-40% with raw LLM prompting. The trade-off is tokens: multi-agent workflows use 2-3x more tokens, but for significant features, the quality improvement justifies the cost.

Human-in-the-Loop Checkpoints

The pattern: Rather than either fully trusting the AI or micromanaging every line, define structured approval gates based on the consequence of the action.

What most developers do instead: Operate in one of two modes. Either review everything line-by-line (treating the AI as fancy autocomplete) or accept large chunks with only a cursory glance. A formatting change and a database schema change get the same level of scrutiny.

Why it matters: Not all changes carry the same risk. A tiered approach gives you speed where it’s safe and control where it matters.

What this looks like in practice: Define personal approval tiers:

• Auto-approve: Formatting, import organisation, adding type annotations
• Quick review: New functions, test additions, single-file refactors
• Careful review: Public API changes, database operations, auth logic
• Full review with plan: Multi-file refactors, new architectural patterns, build/deploy changes

Use small, frequent git commits as checkpoints. If something goes wrong, you can revert to a known-good state without losing everything. Before accepting a change, ask yourself: if this is wrong, what breaks and how hard is it to fix?

Where to Start

You don’t need all nine patterns at once. Start with the ones that address your biggest pain points:

• Code quality issues? Start with structured guardrails and verification loops.
• AI keeps making the same mistakes? Start with persistent memory and context engineering.
• Large diffs that are hard to review? Start with upfront planning and human-in-the-loop checkpoints.
• Spending too much on tokens? Start with deterministic tool delegation and context engineering.
• Not sure if AI is helping? Observability is still largely unsolved, but start by establishing baselines now so you can measure later.

Stop handing the AI the whole problem. Break it down and use the right tool for each step.

This is part of a series on applying patterns from agentic systems to AI-assisted development. See also: The Unix Philosophy for Agentic Coding and Observability for AI-Assisted Development.

That’s from METR’s 2025 study. These were experienced open-source developers working on their own codebases with tools they chose. Their self-assessment was off by over 40 percentage points.

If your perception of AI’s impact is that unreliable, what are you actually measuring?

You Need a Baseline First

If you didn’t measure before AI, measuring with AI won’t work.

You can’t attribute improvements to AI if you don’t know what “before” looked like. Cycle time, deployment frequency, change failure rate, MTTR, value delivered per sprint: these need to exist as baselines before you introduce a new variable. Otherwise you’re guessing, and as the METR study shows, our guesses aren’t great.

I’ve seen teams adopt AI coding assistants and then ask “how do we know it’s helping?” three months later. The real question is six months earlier: “how do we measure effectiveness?” If you didn’t have that defined before AI, you won’t have it now.

What Exists Today

The tooling for observability in AI-assisted development is fragmented. Cost visibility is reasonable. Quality visibility is nearly zero.

Claude Code is the most transparent. It ships with native OpenTelemetry support, tracking tokens, cost, tool calls, and session duration. The /cost command shows real-time spend. /stats visualises daily usage, session history, and model preferences. /insights goes further, analysing your sessions to surface project areas, interaction patterns, and friction points. Commits are auto-tagged with a co-author line, giving you a built-in “was this AI-generated?” marker in your git history. Anthropic provides an official monitoring guide with Grafana dashboard configs and a Docker Compose setup, and the community has built importable dashboards and plugins. The infrastructure for collecting data exists. What to do with it is the harder question.

OpenAI Codex CLI tags commits with a co-author line and supports OTel export for logs and traces. The enterprise dashboard tracks daily users by product, code review completion rates, review priority and sentiment, and session-level message counts. It’s adoption-focused: who’s using what and how much. No quality metrics, no incident correlation, no rework tracking. Individual developers get /status for rate limits but no cost visibility.

Aider has the most configurable commit attribution of any tool (co-author trailers include the model name). But no OTel, no dashboard, no persistent cost history.

GitHub Copilot offers team-level dashboards: acceptance rates, DAU/MAU, feature adoption. It’s oriented toward “is our license worth it?” rather than “is the output good?” No commit tagging.

Cursor exposes very little. A “Year in Code” summary and an “AI Share of Committed Code” metric. No tracing, no commit tagging, no event-level data.

Cline shows per-request cost in the UI (one of its standout features) and supports OTel export at the enterprise tier. No commit tagging.

Amazon Q Developer has the richest built-in analytics dashboard of any tool: acceptance rates, lines of code by feature type, code review counts, per-language breakdowns. But it’s admin-only, subscription-based (no per-token tracking), and publishes to CloudWatch rather than OTel.

Some of us have built our own layers on top. We use Claude Code Usage Monitor to track token usage as a proxy for understanding consumption patterns. It isn’t perfect, isn’t always accurate, but it gives you a feeling for where your usage goes. A few engineers on our teams have personal Grafana dashboards tracking their own AI metrics. But these aren’t centralised, aren’t standardised, and aren’t as useful as they could be.

The picture across the industry: cost visibility is reasonable if you’re willing to set it up. Commit tagging is inconsistent (Claude Code and Codex do it by default, most others don’t). Quality visibility is nearly zero everywhere.

What’s Missing

The gaps fall into three levels: what individual developers need, what teams need, and what organisations need.

For the Individual Developer

No effort distribution. You know how much you spent in tokens. You don’t know where that effort went. Imagine if your AI assistant could tell you: “This week, 40% of your AI time went to test writing, 30% to refactoring, 20% to feature work, 10% to debugging. Your test-writing sessions had the highest acceptance rate. Your debugging sessions cost the most tokens per useful output.” That would let you consciously decide where AI is worth using and where you’re better off working without it.

Limited failure pattern detection. Claude Code’s /insights is the closest thing we have: it analyses sessions and surfaces friction points. That’s a real start, and most other tools don’t offer anything comparable. But it’s still a snapshot of recent sessions, not a long-running trend line. If the AI keeps making the same category of mistake (wrong import paths, ignoring your test conventions, using a deprecated API), you want something that surfaces “you’ve corrected the AI on import paths 12 times this month” and suggests adding it to your CLAUDE.md. Some people maintain a manual lessons-learned.md where they log AI mistakes. It works, but it’s ad hoc.

No context effectiveness feedback. CLAUDE.md files are checked in, reviewed in PRs, and engineered for effectiveness over time, much like prompts. The feedback loop exists but it’s manual and slow. You notice the AI getting something wrong, update the file, and see if it improves. What’s missing is the measurement that closes the loop: did that change actually improve output quality, or did it just feel like it did? The METR perception gap applies here too.

For the Team

No aggregate failure patterns. If three engineers on the same team are all hitting the same AI failure mode, that’s not three individual problems. It’s a systemic context gap: a missing architectural convention, an undocumented pattern, a guardrail that should exist but doesn’t. No tool surfaces this today.

No RCA correlation. Claude Code tags commits with a co-author line. That’s the “was this AI-generated?” link in the RCA chain. But other tools don’t do this consistently. And even with the tag, nobody is aggregating that data: correlating AI-tagged commits with incident rates, rework rates, or review times over time. Traditional RCA follows a clear chain (incident → deployment → commit → PR → review → root cause). AI adds a question to that chain: was the reviewer’s miss caused by a large AI-generated diff? Was the AI missing context it should have had? Is this a known AI weakness that should be in the team’s guardrails?

The velocity flatline problem. We’ve seen this firsthand. Teams get faster with AI. Then velocity flattens. Not because AI stopped helping, but because teams redirected the extra capacity to paying off debt or solving problems they found interesting. That’s not necessarily bad, but if you’re not tracking what work goes where, you can’t tell the difference between “team is investing in sustainability” and “team is coasting.”

The fix we found: track work against cards. Measure total value delivered, not just pace. Make sure the extra capacity from AI shows up as increased value, not just different work. This is a process fix, not a tooling fix. No observability tool surfaces this today.

For the Organisation

No cross-team maturity view. Some teams will be excellent at AI-assisted development. Others will struggle. As a CTO, you need to know which is which, and more importantly, what the effective teams are doing differently. Are they better at context engineering? More disciplined about review? Today, finding this out requires manual investigation.

No automated “are we improving?” picture. This is the hardest gap. Drawing a full picture of whether an engineering organisation is improving has always required someone to build that view manually. AI hasn’t changed that. It’s just added another variable.

The data exists. Commits are tagged. Tickets track value. CI tracks quality. AI tools track cost and usage. But nobody is stitching them into a coherent picture that answers: “Is AI helping us deliver more value, or is it making us feel faster while quality degrades?”

What We’d Like to See

Here’s what I wish existed:

AI timesheets. Not for billing. For self-awareness. Show me where my AI time goes, which task types have the best return, and where I’m burning tokens for low value. Let me compare across weeks and see trends.

Automated RCA tagging. Correlate AI-tagged commits with downstream incidents, reverts, and rework. Not to blame the tool, but to know where to invest in better review, context, or guardrails.

Context effectiveness scoring. When I change my CLAUDE.md, show me whether output quality improved for the task types I was targeting. Even a rough signal (fewer corrections needed, lower rework rate) would be valuable.

Failure pattern aggregation. Surface repeated AI mistakes at the team level. If the same failure shows up across engineers, flag it as a context gap, not an individual problem.

The org-wide picture, stitched together. Combine git data, ticket data, CI data, and AI usage data into a view that answers: are we delivering more value? Is quality holding? Where should we invest next?

Questions for Solution Builders

If you’re building in this space, here are the questions I’d want answered:

1. Can the “are we improving?” picture be automated? The data is there (git, tickets, CI, AI usage). Can you stitch it together without someone manually maintaining a dashboard? Can you infer value delivery trends from data that already exists?

2. How do you measure context effectiveness without controlled experiments? A/B testing CLAUDE.md configurations isn’t practical in real workflows. What proxy signals can tell us whether a context change helped?

3. What does a useful AI timesheet look like? Not session-level token counts, but task-level effort distribution. How do you classify AI sessions by task type without requiring the developer to manually tag them?

4. How do you surface failure patterns across a team? Individual correction patterns are noisy. Aggregate patterns are signal. What’s the right level of abstraction?

5. How do you separate “AI made us faster” from “we redirected capacity”? Velocity metrics alone can’t tell you this. What combination of signals can?

6. How do you handle the perception gap? Developers believe they’re faster. Measurement sometimes shows otherwise. How do you present this data in a way that’s constructive rather than demoralising?

These aren’t rhetorical questions. If you’re building tools in this space, I’d like to hear your answers.

This is the second post in a series on applying patterns from agentic systems to everyday AI-assisted development. The first, The Unix Philosophy for Agentic Coding, covers deterministic tool delegation.

There’s a better way. One that’s cheaper, more predictable, and already well understood. It’s the Unix philosophy, applied to how we work with AI.

The Pattern

The Unix philosophy boils down to: do one thing well, compose small tools, let the shell orchestrate. When you work with an AI coding agent, the agent is the shell.

Here’s how I think about it:

1. Break the problem down. Don’t hand the agent a big, vague goal. Decompose it into sub-problems.
2. If a tool exists, use it. Refactoring, formatting, linting, deployment: these are solved problems. Don’t ask the AI to reinvent them.
3. If no tool exists, build one. A small, deterministic script is better than an LLM making judgment calls where none are needed.
4. The agent orchestrates. It decides what to do, in what order, with which tools. That’s where its intelligence adds value.

The principle is simple: don’t let AI make decisions it doesn’t need to make.

Every unnecessary decision is a degree of freedom. Every degree of freedom is an opportunity for the model to get something wrong, burn tokens, and produce a result you can’t reproduce.

What Goes Wrong Without This

When you ask an AI agent to do something a deterministic tool already handles, you get:

• Inconsistency. LLMs aren’t deterministic. Run the same prompt twice, get different results. A tool gives you the same output every time.
• Wasted tokens. Generating 200 lines of reformatted code costs tokens. Running Prettier or Ruff costs nothing.
• More failure modes. The model might miss edge cases a dedicated tool handles by design. A refactoring tool knows about downstream dependencies. An LLM might not.
• Slower feedback loops. Generating code, reviewing it, finding the error, regenerating: that cycle is slower than calling a tool that gets it right the first time.

Examples

Refactoring

I want to rename a method. The method is used across dozens of files.

The naive approach: ask the agent to read the codebase, find all references, and rewrite them. The agent will try. It might miss some. It might introduce a formatting inconsistency along the way. You’ll spend time reviewing a diff that’s harder to trust.

The better approach: the agent calls IntelliJ’s refactoring tools via MCP. One command. Every reference updated. Downstream dependencies handled. No formatting changes. No guesswork.

Refactoring is a solved problem. I wouldn’t ask a teammate to do a manual find-and-replace across a codebase. I wouldn’t ask an AI agent to either.

Analysing CSV Data

I have a set of CSVs I need to extract insights from.

The naive approach: hand the files to the agent and ask it to read, validate, extract, and summarise everything. The agent will try. It might misparse a column, silently drop malformed rows, or hallucinate a trend that isn’t there. You won’t know unless you check every step. Large CSVs make this worse. Hundreds of thousands of rows won’t fit in a context window, and even if they did, you’re burning tokens on data the model doesn’t need to see. The agent doesn’t know which rows matter until it’s processed all of them.

The better approach: build a small CLI that pre-processes the data first. Validate schemas, flag missing values, confirm row counts, filter to the relevant subset, compute the aggregations that don’t need intelligence. This is deterministic work. Then pass the clean, reduced output to the agent for the part that actually needs judgment: identifying patterns and summarising insights.

No tool existed for this specific validation, so I asked the agent to build one. That’s the pattern. Build the tool, then use the tool. The agent wrote a script I can run repeatedly with predictable results. Now it’s free to focus on what it’s good at.

Code Formatting

I want my code to follow our team’s style guide.

The naive approach: include the style guide in the prompt and ask the agent to follow it. It will mostly comply. It will sometimes get creative (especially as context fills up). You’ll find inconsistencies across files that are annoying to track down.

The better approach: let the agent write code however it wants, then run Prettier, Black, Ruff, or ESLint. Zero ambiguity. The agent doesn’t need to think about formatting at all, which means fewer tokens spent and fewer decisions that could go wrong.

Skills, Hooks, and Tools

If you use Claude Code, you’ll know about skills (composable prompt-driven capabilities) and hooks (event-driven automation). These are the wiring. But wiring without workers doesn’t accomplish much.

A good skill is composable. A great skill is composable and delegates to deterministic tools instead of taking on responsibilities it doesn’t need. If a skill invokes a CLI tool, an API, or a build system instead of asking the LLM to reason through a solved problem, that skill will be faster, cheaper, and more reliable.

The same applies beyond Claude Code. Cursor rules, Windsurf workflows, any AI assistant: the pattern holds. Build your workflows so the AI orchestrates tools, not replaces them.

The Bigger Picture

This isn’t just about code formatting and refactoring. The same principle applies to deployment pipelines, database migrations, CI/CD workflows, building CLIs for business operations. Anywhere a deterministic tool can guarantee a correct result, use it. Reserve the LLM for the parts that genuinely need judgment: understanding intent, choosing an approach, reasoning about trade-offs, writing novel logic.

Not every problem needs this treatment. For exploratory work, prototyping, or genuinely novel problems, letting the agent roam is the right call. But for the repeatable parts of your workflow, reach for a tool.

The best AI workflows I’ve built look like Unix pipelines. Small, focused tools. A smart orchestrator composing them. The AI’s value isn’t in doing everything. It’s in knowing what to do and calling the right tool to do it.

Thanks to Carmen Mardiros whose talk at Data Engineers London helped crystallize this thinking.

I’ve written about intelligent Engineering principles and the skills needed to build with AI. But I kept getting the same question: “How do I actually set this up on a real project?”

This post answers that question. I’ll walk through the complete setup, using a real repository as a worked example. Not a toy project. Not a weekend experiment. A codebase with architectural decisions, test coverage, documentation, and a clear development workflow.

Here’s what it looks like in action:

The intelligent Engineering Stack

Before diving into details, here’s the mental model I use. intelligent Engineering isn’t one thing. It’s layers that enable each other:

This diagram shows Claude Code’s primitives. Other AI assistants have different building blocks: Cursor has rules and .cursorrules, Windsurf has Cascade workflows. The layers matter more than the specific implementation.

The screencast showed the workflow. The rest of this post explains what makes it work, layer by layer from top to bottom.

The Two Phases of intelligent Engineering

Shaping AI is preparation. You define agentic workflows, set up tooling, provide context, and build a prompt library. Context includes coding guidelines, architecture patterns, and deployment patterns. This is the work before you write code.

Leading AI is execution. You move through the development cycle: ideation, research, feature specs, architecture, stories, code, test, deploy. This is the work of building software, with AI as a collaborator.

The screencast showed Leading AI in action. But Leading AI only works well when Shaping AI is done right. Most of this post focuses on Shaping AI, the setup that makes everything else possible.

We’ve written more about this framework on the Sahaj website.

Level 4: The Workflow

This workflow mirrors how many teams already work: pick up a card, groom it, discuss the approach, TDD the implementation. Your team might work differently. intelligent Engineering is about shaping AI to match your existing practices, not adopting a prescribed workflow. The stack exists to encode how your team works. You should be shaping AI, not letting it shape you. This example shows one approach; yours will look different.

Here’s what the screencast showed, step by step:

1. Pick Up Work

> /pickup

## Available Stories by Priority

### P0 - Critical
- #23 - S01.3: Profile Completion [module:customer]

Which card would you like to pick up?

2. Design Discussion

Here’s what the design discussion looks like:

I've read #23 and the customer domain docs. Here's my understanding:

**What we're building:**
- Profile completion endpoint (PUT /api/v1/customers/{id}/profile)
- Validation: name, DOB (18+), SSN format, address
- SSN encryption before storage

**Proposed approach:**
1. Add ProfileRequest DTO with validation annotations
2. Extend CustomerService with completeProfile method
3. Add encryption service for SSN

**Questions:**
- Should SSN validation reject obvious invalid patterns (000-00-0000)?
- Should we allow profile updates after initial completion?

This is Leading AI in action. The AI proposes, the human validates and refines. Agreement on approach before code.

3. TDD Implementation

Test-Driven Development (TDD) is a discipline where you write tests before code. The cycle is Red-Green-Refactor: write a failing test (red), write minimum code to pass (green), clean up (refactor). Kent Beck’s Test-Driven Development by Example is the canonical reference.

With agreement on approach, the TDD skill takes over:

Running in interactive mode...

RED: Writing test for profile completion with valid data

The TDD skill supports multiple review modes that control when you review AI’s work. Interactive mode pauses after each red-green cycle; batch and autonomous modes give more flow with less oversight.

The AI writes a failing test:

@Test
void shouldCompleteProfile_whenAllFieldsValid() {
    var request = aProfileRequest()
        .withFirstName("John")
        .withLastName("Doe")
        .withDateOfBirth(LocalDate.of(1990, 1, 15))
        .withSsn("123-45-6789")
        .build();

    var result = customerService.completeProfile(customerId, request);

    assertThat(result.status()).isEqualTo(PROFILE_COMPLETE);
}

And runs it:

Test written. Running...
❌ FAILED: completeProfile method doesn't exist

RED confirmed. Ready for GREEN?

This continues through each acceptance criterion with small commits after each cycle.

4. Context Updates

If you use /pickup to start work, context updates happen automatically when the story completes. The /update-context command is for when you need to update docs outside the standard workflow:

> /update-context

Recent changes:
- Added ProfileRequest DTO
- Added completeProfile to CustomerService
- Added SSN encryption

Context docs to review:
- docs/context/domain/customer.md (profile rules)
- docs/context/modules/customer-module.md (new endpoint)
- docs/context/current-state.md (mark feature built)

Proceed with updates?

5. Push to Main

Trunk-based development. Small commits. Direct to main.

git push origin main

Story auto-closes via the Closes #23 footer in the final commit.

That’s the workflow. Now let’s understand what makes it possible.

Level 3: The Interaction Layer

This is how you interact with the AI during development. The examples use Claude Code primitives, but the concepts transfer to other tools:

Claude Code organizes these into distinct primitives: commands, skills, and hooks. Each serves a different purpose.

Design Principles

Whether you use Claude Code, Cursor, or another tool, these principles apply:

Description quality is critical. AI tools use descriptions to discover which skill to activate. Vague descriptions mean skills never get triggered. Include what the skill does AND when to use it, with specific trigger terms users would naturally say.

# Bad
description: Helps with testing

# Good
description: Enforces Red-Green-Refactor discipline for code changes.
             Use when implementing features, fixing bugs, or writing code.

Single responsibility. Each command or skill does one thing. /pickup selects work. /start-dev begins development. Combining them makes both harder to discover and maintain.

Give goals, not steps. Let the AI decide specifics. “Sort by priority and present options” beats a rigid sequence of exact commands. The AI can adapt to context you didn’t anticipate.

Include escape hatches. “If blocked, ask the user” prevents infinite loops. AI will try to solve problems; give it permission to ask for help instead.

Progressive disclosure. Keep the main instruction file concise. Put detailed references in separate files that load on-demand. Context windows are shared: your skill competes with conversation history for space.

Match freedom to fragility. Some tasks need exact steps (database migrations). Others benefit from AI judgment (refactoring). Use specific scripts for fragile operations; flexible instructions for judgment calls.

Test across models. What works with a powerful model may need more guidance for a faster one. If you switch models for cost or speed, verify your skills still work.

Commands

Commands are user-invoked. You type /pickup and something happens.

Here’s the command set I use:

Each command is a markdown file in .claude/commands/ with instructions for the AI:

# Pick Up Next Card

You are helping the user pick up the next prioritized story.

## Instructions

1. Fetch open stories using GitHub CLI
2. Sort by priority (P0 first, then P1, P2)
3. Present options to the user
4. When selected, assign the issue
5. Show issue details to begin work

The /tour command walks through project architecture, module structure, coding conventions, testing approach, and domain glossary. It turns context docs into an interactive onboarding experience.

Skills

Skills are model-invoked. The AI activates them automatically based on context. If I ask to “implement the registration endpoint,” the TDD skill activates without me saying /tdd.

The TDD skill is the one I use most:

Trigger: User asks to implement something, fix a bug, or write code

Workflow:

1. RED: Write a failing test, run it, confirm it fails
2. GREEN: Write minimum code to pass, run tests, confirm green
3. REFACTOR: Clean up while keeping tests green
4. COMMIT: Small commit with issue reference

Review modes control how much human oversight:

I typically use interactive mode for unfamiliar code and batch-ac mode for well-understood patterns. I mostly use batch-story and autonomous modes for demos, though they’d suit repetitive work with well-established patterns.

The review skill provides structured feedback:

## Code Review: normal mode

### Blockers (0 found)

### Warnings (2 found)
1. **CustomerService.java:45** Method exceeds 20 lines
   - Consider extracting validation logic

### Suggestions (1 found)
1. **CustomerServiceTest.java:112** Test name could be more specific

### Summary
- Blockers: 0
- Warnings: 2
- Suggestions: 1
- **Verdict**: NEEDS ATTENTION

The autonomous TDD mode uses this skill with configurable thresholds. “Strict” interrupts on any finding. “Relaxed” only stops for blockers.

Hooks

Hooks are event-driven. They run shell commands or LLM prompts at specific lifecycle events: before a tool runs, after a file is written, when Claude asks for permission.

Example: auto-format with Prettier after every file write:

{
  "hooks": {
    "PostToolUse": [{
      "matcher": "Write|Edit",
      "hooks": [{
        "type": "command",
        "command": "npx prettier --write \"$FILE_PATH\""
      }]
    }]
  }
}

The credit-card-lending project doesn’t use hooks yet. They’re next on the list.

Other Primitives

Claude Code has additional constructs I haven’t used in this project:

Start with commands, skills, and context docs. Add the others as your needs grow.

Level 2: Context Documentation

Context is what the AI knows about your project. I’ve seen teams underinvest here. They write a README and call it done, then wonder why AI assistants keep making the same mistakes.

What’s missing is your engineering culture. The hardest part isn’t the tools, it’s capturing what your team actually does. For example, code reviews are hard because most time goes to style, not substance. “Why isn’t this using our logging pattern?” “We don’t structure tests that way here.” Without codification, AI applies its own defaults. The code might work, but it doesn’t feel like your code.

When you codify your team’s preferences, AI follows YOUR patterns instead of its defaults. Style debates shift left: instead of the same argument across a dozen pull requests, you debate once over a document. Once the document reflects consensus, it’s settled.

What to Document

I’ve settled on this structure:

The credit-card-lending project extends this with integrations.md (external systems) and metrics.md (measuring iE effectiveness). Adapt the structure to your domain’s needs.

These docs exist for both AI and human consumption, but discoverability matters. New team members shouldn’t have to hunt through docs/context/ to understand what exists. The credit-card-lending project solves this with a /tour command: run it and get an AI-guided walkthrough covering architecture, conventions, testing, and domain knowledge. This transforms static documentation into an interactive onboarding flow. Context docs become working tools, not forgotten reference material.

Context Doc Anatomy

Every context doc starts with “Why Read This?” and prerequisites:

# Testing Strategy

## Why Read This?

TDD principles, test pyramid, and testing tools.
Read when writing tests or understanding the test approach.

**Prerequisites:** conventions.md for code style
**Related:** domain/ for business rules being tested

---

## Philosophy

We practice Test-Driven Development as our primary approach.
Tests drive design and provide confidence for change.

This helps AI tools (and humans) know whether they need this file and what to read first.

Dense facts beat explanatory prose. Compare:

“Our testing philosophy emphasizes the importance of test-driven development. We believe that writing tests first leads to better design…”

vs.

“TDD: Red-Green-Refactor. Tests before code. One assertion per test. Naming: should{Expected}_when{Condition}.”

The second version is what AI tools need. Save the narrative for human-focused documentation.

Living Documentation

Stale documentation lies confidently. It states things that are no longer true. You write tests to catch broken code. Your documentation needs the same capability.

The credit-card-lending project handles this two ways:

1. Definition of Done includes context updates: Every story card lists which context docs to review. The AI won’t let you forget. You can bypass it by working without your AI pair or deleting the prompt, but the default path nudges you toward keeping docs current.
2. Drift detection: A /check-drift command compares docs against code

The second point catches what the first misses. I’ve seen projects where features get built but current-state.md still shows them as planned. Regular drift checks catch this before it causes confusion.

Patterns for Teams

The examples above work within a single repository. At team and org level:

Shared context repository: A company-wide repo with organization-level conventions, security requirements, architectural patterns. Each project references it but can override.

Team-level customization: Team-specific CLAUDE.md additions for their domain, their tools, their workflow quirks.

Prompt library: Reusable prompts for common tasks. “Review this PR for security issues” with the right context attached.

Level 1: Foundation

The foundation is what the AI sees when it first encounters your project.

CLAUDE.md

This is your project’s instruction manual for AI assistants. It goes in the repository root and contains:

• Project context: What this is, what it does
• Git workflow: Commit conventions, branching strategy
• Context file references: Where to find domain knowledge, conventions, architecture
• Tool-specific instructions: Commands, scripts, common tasks

Here’s an excerpt from the credit-card-lending CLAUDE.md:

# CLAUDE.md

## Project Context
Credit card lending platform built with Java 25 and Spring Boot 4.
Modular monolith architecture with clear module boundaries.

## Git Workflow
- Trunk-based development: push to main, no PRs for standard work
- Small commits (<200 lines) with descriptive messages
- Reference issue numbers in commits

## Context Files
Read these before working on specific areas:
- `docs/context/overview.md` - Architecture and module structure
- `docs/context/conventions.md` - Code standards and patterns
- `docs/context/testing.md` - TDD principles and test strategy

CLAUDE.md is dense and factual, not explanatory. It tells the AI what to do, not why. The “why” lives in context docs.

Project Structure

Structure matters because AI tools use file paths to understand context. I’ve found this layout works well:

project/
├── CLAUDE.md                    # AI instruction manual
├── .claude/
│   ├── commands/                # User-invoked slash commands
│   └── skills/                  # Model-invoked capabilities
├── docs/
│   ├── context/                 # Dense reference documentation
│   │   ├── overview.md
│   │   ├── conventions.md
│   │   ├── testing.md
│   │   └── domain/
│   ├── wiki/                    # Narrative documentation
│   └── adr/                     # Architectural decisions
└── src/                         # Your code

The separation between context/ (for AI consumption) and wiki/ (for humans) is intentional. Context docs are dense facts. Wiki pages explain concepts with diagrams and narrative. ADRs (Architectural Decision Records) capture why significant decisions were made. This context prevents future teams from wondering “why did they do it this way?”

Takeaways

The credit-card-lending repository demonstrates everything discussed above. Here’s what I learned applying it.

What Worked

Small batches: Most commits are under 100 lines. This makes review meaningful and rollbacks clean.

Context primacy: The AI reads conventions.md before writing code. It knows our test naming patterns, package structure, and error handling approach without me repeating it.

TDD skill with review modes: Interactive mode for complex validation logic. Batch-ac mode for straightforward CRUD operations.

Living documentation: Every completed story updates current-state.md. I know what’s built by reading one file.

What We Learned

Context docs need maintenance: Early on, I’d update code without updating context docs. The AI would then generate code following outdated patterns. The /check-drift command catches this now.

Skills are better than scripts: I started with bash scripts for workflows. Moving to skills let the AI adapt to context instead of following rigid steps.

Design discussion matters: Agreeing on approach before coding feels slow. In reality, it saves rework.

Getting Started

Ready to try this? Here’s a path:

If You’re Starting Fresh

1. Create CLAUDE.md with your project context
2. Add docs/context/conventions.md with your coding standards
3. Start with one command: /start-dev for TDD workflow
4. Add context docs as you need them

If You Have an Existing Project

1. Create CLAUDE.md capturing how you want the project worked on
2. Document your most important conventions
3. Add the /update-context command so documentation stays current
4. Gradually expand context as you work

Try It Yourself

Clone the example repository and explore:

git clone https://github.com/javatarz/credit-card-lending
cd credit-card-lending

Run /tour to get an interactive walkthrough of the project structure, setup, and key concepts. Then try /pickup to see available work or /start-dev to see TDD in action.

The branch blog-ie-setup-jan2025 contains the exact state referenced in this post.

What’s Next

If you try this approach, I’d like to hear what works and what doesn’t. The practices here evolved from experimentation. They’ll keep evolving.

Credits

The intelligent Engineering framework was developed in collaboration with Anand Iyengar and other Sahajeevis. It was originally published on the Sahaj website.

In my previous post on intelligent Engineering principles, I outlined the ideas that guide how I build software with AI. Since then, I’ve had people ask: “Where do I start? What skills should I build first?”

This post answers that: a map of the skills that make up intelligent Engineering, organised into a learning path you can follow whether you’re an individual contributor looking to level up or a tech leader building your team’s AI fluency.

What is intelligent Engineering?

intelligent Engineering is a framework for integrating AI across the entire software development lifecycle, not just code generation.

Writing code represents only 10-20% of software development effort. The rest is research, analysis, design, testing, deployment, and maintenance. intelligent Engineering applies AI across all of these stages while keeping humans accountable for outcomes.

I’ve already written about the five core principles in detail. This post focuses on the skills that make those principles actionable.

The Skill Map

Master the skills at each stage before moving to the next. Skipping ahead creates gaps that AI will expose.

1. Foundations

The 2025 DORA report confirmed what many suspected: AI amplifies your existing capability, magnifying both strengths and weaknesses.

If your fundamentals are weak, AI won’t fix them. It will make the cracks more visible, faster.

This map assumes you already have solid computer science fundamentals: data structures, algorithms, and an understanding of how systems work (processors, memory, networking, databases, etc.). AI doesn’t replace the need to know these.

Version control fluency

Git workflows, meaningful commits, safe experimentation with branches. AI generates code quickly. If you can’t safely integrate and roll back changes, you’ll spend more time cleaning up than you save.

How to build this:

• If you haven’t used branches and pull requests regularly, start a side project that forces you to
• Read Pro Git (free online) - chapters 1-3 cover the essentials
• Learn git worktrees - you’ll need them for multi-agent workflows in the Advanced section

Testing fundamentals

The test pyramid still applies. Unit, integration, end-to-end. AI can generate tests, but knowing which tests matter, when to push tests up or down the pyramid, and reviewing their quality is your job. Build intuition for what belongs at each layer.

How to build this:

• Practice writing tests before code (TDD) on a small project
• Read Test-Driven Development: By Example by Kent Beck, the foundational TDD book
• Read Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pryce for TDD in practice
• Apply Martin Fowler’s test pyramid rule: if a unit test covers it, don’t duplicate at higher levels. Push tests down: unit test business logic, integration test service interactions, end-to-end only for critical user paths

Code review discipline

You’ll review more code than ever. AI-generated code often looks plausible but handles edge cases incorrectly. Strengthen your eye for subtle bugs.

What to watch for in AI-generated code:

• Security vulnerabilities: SQL injection, unsafe data handling, hardcoded secrets. AI often generates patterns that work but aren’t secure.
• Edge cases: Null handling, empty collections, boundary conditions. AI tends to handle the happy path well but miss edge cases.
• Business logic errors: AI can’t understand your domain. Verify that the code does what the business actually needs, not just what the prompt described.
• Architectural violations: Does the code respect your layer boundaries? Does it follow your ADRs? AI doesn’t know your architectural constraints unless you tell it.
• Code smells: Duplicated logic, overly complex methods, inconsistent patterns. AI doesn’t always match your codebase conventions.
• Hallucinated APIs: Functions or methods that look real but don’t exist. Always verify imports and dependencies.

How to build this:

• Review pull requests on open source projects you use
• Read Code Review Guidelines from Google’s engineering practices
• Practice the “trust but verify” mindset: assume AI code needs checking, not approval

Code quality intuition

Can you recognize maintainable, clean code vs technically-correct-but-messy? AI generates code fast. If you can’t tell good from bad, you’ll accept garbage that costs you later.

How to build this:

• Read Clean Code by Robert Martin
• Refactor old code you wrote, or practice on clean code katas - notice what makes code hard to change

Documentation practices

Documentation becomes AI context. Quality documentation into the system means quality AI output. Poor docs mean the AI hallucinates or makes wrong assumptions.

How to build this:

• Document a project you’re working on as if a new teammate needs to understand it
• Read Docs for Developers for practical guidance

Architecture understanding

Data flow, component boundaries, dependency management. AI tools need you to describe constraints clearly. If you don’t understand the architecture, you can’t provide good context.

How to build this:

• Draw architecture diagrams for systems you work with
• Read Fundamentals of Software Architecture by Richards and Ford for trade-offs and patterns
• Read Designing Data-Intensive Applications by Kleppmann for distributed systems and data architecture
• For microservices specifically, read Building Microservices by Sam Newman

2. AI Interaction

The skills specific to working with AI systems. You’re learning to communicate with a system that’s capable but context-limited, confident but sometimes wrong.

Prompt engineering basics

Specificity matters. Vague requests get vague results.

Bad prompt:

Write a function to parse dates

Good prompt:

Write a Python function that:
- Parses ISO 8601 date strings (e.g., "2025-12-31T14:30:00Z")
- Handles timezone offsets
- Returns None for invalid input
- Include docstring and type hints

The difference isn’t cleverness - it’s precision.

Key techniques:

Few-shot example:

Convert these function names from camelCase to snake_case:

Example 1: getUserById -> get_user_by_id
Example 2: validateEmailAddress -> validate_email_address

Now convert: fetchAllActiveUsers

Chain of thought example:

Debug this function. Think step-by-step:
1. What is this function supposed to do?
2. Trace through with input X - what happens at each line?
3. Where does the actual behavior differ from expected?
4. What's the fix?

How to build this:

• Spend a week being deliberate about prompts. Write down what you asked, what you got, and what you wish you’d asked.
• Read Anthropic’s Prompt Engineering Guide
• Reference promptingguide.ai for comprehensive techniques

Context engineering

A clever prompt won’t fix bad context. Context engineering is about curating what information the model sees: project constraints, coding standards, relevant examples, what you’ve already tried.

This is the 80% of the skill. Prompt engineering is maybe 20%.

I’ve written a detailed guide on this: Context Engineering for AI-Assisted Development.

How to build this:

• Create a project-level context file (e.g., CLAUDE.md) for your current codebase
• Add coding standards, architectural constraints, common patterns
• Notice when AI output improves because of better context

Understanding model behaviour

You don’t need to become an ML engineer, but knowing the basics helps.

What to understand:

Model strengths (from my experience):

These observations come from my own work. Models evolve quickly - what’s true today may change next quarter.

How to build this:

• Try the same task with different models. Note where each excels.
• Read model release notes when new versions come out
• Track which models work best for your common tasks

Understanding tool behaviour

Here’s something that trips people up: the same model behaves differently in different tools.

Cursor’s Claude is not the same as Claude Code’s Claude is not the same as Windsurf’s Claude. Why? Each tool wraps the model with its own system prompt.

This means: instructions that work well in Claude Code might not work the same in Cursor, even with the same underlying model. The tool’s system prompt and context injection change the behavior.

How to build this:

• Try the same prompt in multiple tools. Notice the differences.
• Read your tool’s documentation on how it manages context
• Understand what your tool’s system prompt optimizes for (coding, conversation, etc.)

3. Workflow Integration

Making AI a standard part of how you build software, not a novelty you occasionally use.

Tool configuration

Configure your AI tools for your team’s context. This isn’t a one-time setup. Rules files need tuning. Context evolves. Tools update frequently.

Each tool has its own configuration mechanism:

• Claude Code uses CLAUDE.md files
• Cursor uses rules
• Windsurf uses memories

Instructions that work in one tool won’t transfer directly to another because system prompts differ.

How to build this:

• Document your configuration so teammates can get productive quickly
• Review and update configuration monthly as tools evolve

Specs-before-implementation

Define what to build before AI generates code. AI generates code that matches a spec well. It struggles to determine what the spec should be.

Write the spec first - acceptance criteria, edge cases, constraints. Then let AI implement.

How to build this:

• Practice writing specs for features before touching code
• Include: what it should do, what it shouldn’t do, edge cases to handle

Test-driven mindset with AI

Write tests first. Let AI implement to pass them. This flips the usual flow: instead of “generate code, then test it”, you “define the contract, then fill it in.”

The tests become your spec. When AI has an executable target (tests that must pass), it produces better code than when interpreting prose requirements.

How to build this:

• Try TDD on a small feature: write failing tests, then ask AI to make them pass
• Review the generated code - does it just satisfy the tests or is it actually good?

Human review gates

AI-generated code requires the same (or stricter) review as human-written code. Build the habit of treating AI output like code from a confident junior developer: often correct, sometimes subtly wrong, occasionally completely off base.

How to build this:

• Set a personal rule: no AI-generated code merged without reviewing every line
• Track your AI acceptance rate. If you’re accepting >80% without modification, you might be over-trusting.

Small batches

Generate less, review more. A 1000-line AI diff is harder to review than a 100-line one. Work in small chunks. Commit often.

How to build this:

• Break tasks into steps that produce <200 lines of change
• Commit after each step passes review

Quality guardrails

Integrate linting, static analysis, and security scanning into your workflow. These catch issues AI introduces. Shift left. Catch problems early.

How to build this:

• Set up pre-commit hooks for linting and formatting
• Add security scanning to CI (e.g., Snyk, Semgrep)

Living documentation

Documentation updated atomically with code changes. When code changes, docs change in the same commit. This keeps your AI context current.

How to build this:

• Include doc updates in your definition of done
• Review PRs for documentation staleness

4. Advanced / Agentic

Skills for autonomous AI workflows. These are powerful but risky - more autonomy needs stronger guardrails.

Agentic workflow design

Tools like Claude Code, Cursor, and Windsurf can run shell commands, edit files, and chain actions. Know what your tool can do and design workflows that leverage it.

How to build this:

• Start with supervised agents - review each step before allowing the next
• Read Claude Code’s GitHub Actions integration for CI/CD examples

Task decomposition

Break complex work into subtasks an agent can handle. Good decomposition is a skill in itself. Too big and the agent loses focus. Too small and you spend all your time orchestrating.

How to build this:

• Practice breaking features into agent-sized tasks (~30 min of work each)
• Notice which decompositions lead to better agent output

Guardrails for agents

More autonomy needs stronger guardrails. Sandboxing, approval gates, rollback procedures. Agents make mistakes. Build systems that catch them.

How to build this:

• Never give agents write access to production
• Implement approval gates for destructive operations

Engineering culture codification

Turn your team’s standards, patterns, and guidelines into structured artifacts that AI can use. This is how you scale intelligent Engineering beyond individuals.

When you document coding standards, architectural patterns, and review checklists in a format AI can consume, every team member (and AI tool) operates from the same playbook.

How to build this:

• Start with a CLAUDE.md (or equivalent) that captures your team’s conventions
• Add architectural decision records (ADRs) that AI can reference

Multi-agent orchestration

Running parallel agents (e.g., using git worktrees). Coordinating results. This is emerging territory.

How to build this:

• Try running two agents on independent tasks
• Notice coordination challenges and develop patterns for handling them

CI/CD integration

Running AI reviews on pull requests. Automated code analysis. Scheduled agents for maintenance tasks.

How to build this:

• Set up Copilot code review or similar on your repo
• Start with comment-only (no auto-merge) until you trust it

Learning Paths

Not everyone starts from the same place.

For Developers New to AI Tools

Start here: Foundations + AI Interaction basics

1. Get comfortable with one AI tool. GitHub Copilot is a good starting point for its low cost and tight editor integration. For open source alternatives, try Aider or OpenCode.
2. Spend 2-4 weeks using it for completion and simple generation.
3. Practice prompting: be specific, iterate, learn what works.
4. Move to a more capable tool (Claude Code, Cursor, Windsurf) once you’re comfortable.
5. Build your first context file.

Expected ramp-up: 4-8 weeks to feel productive.

For Developers Experienced With AI

Start here: Workflow Integration + Advanced

1. Audit your current workflow. Where are you using AI effectively? Where are you over-trusting?
2. Strengthen context engineering. Create comprehensive project context files.
3. Set up guardrails: linting, security scanning, review checklists.
4. Experiment with agentic workflows under supervision.
5. Integrate AI into CI/CD.

Expected ramp-up: 2-4 weeks to significantly improve your workflow.

For Tech Leaders Building Team Capability

Whether you’re a Tech Lead, Engineering Manager, Principal Engineer, or anyone else responsible for growing your team’s capability, this section is for you.

Start here: The 2025 DORA AI Capabilities Model

The report identified seven practices that amplify AI’s positive impact:

1. Clear AI stance: Establish expectations for how your team uses AI.
2. Healthy data ecosystem: Quality documentation enables quality AI outputs.
3. Strong version control: Rollback capability provides a safety net for experimentation.
4. Small batches: Enable quick course corrections.
5. User-centric focus: Clear goals improve AI output quality.
6. Quality internal platforms: Standardised tooling scales AI benefits.
7. AI-accessible data: Make context available to AI tools.

Actions:

1. Assess your team against these practices. Where are the gaps?
2. Don’t change everything at once. Introduce AI at one delivery stage at a time.
3. Expect a learning curve: 2-4 weeks of reduced productivity before gains appear.
4. Invest in guardrails before acceleration.
5. Measure impact with DORA metrics: deployment frequency, lead time, change failure rate, time to restore.

Common Pitfalls

Starting with advanced tools: If you skip fundamentals, you’ll produce more code, faster, with worse quality. The problems compound.

Ignoring context engineering: Most teams spend all their energy on prompt engineering. Context engineering matters far more. Good context makes mediocre prompts work; perfect prompts can’t fix missing context. And context scales: set it up once, benefit every interaction.

Over-trusting AI: “The AI suggested it” is not an acceptable answer in a post-mortem. You’re accountable for what ships.

Under-trusting AI: Some developers refuse to adopt AI tools, treating them as a passing fad. The productivity gap is real. Healthy skepticism is fine, but refusing to engage is risky. For tech leaders: DORA’s research on AI adoption shows that addressing anxieties directly and providing dedicated exploration time significantly improves adoption.

No guardrails: AI makes it easy to move fast. Without automated quality checks, you’ll ship bugs faster too. Smarter AI needs smarter guardrails. If you don’t have linting, security scanning, and CI checks, add them before increasing your AI usage. For legacy codebases without tests, start with characterization tests to capture current behaviour before refactoring. Michael Feathers’ Working Effectively with Legacy Code is the definitive guide here. AI can accelerate this process, but verify every generated test passes against the real system without any changes to production code.

Confusing model and tool behaviour: When AI output is wrong, is it the model’s limitation or the tool’s system prompt? Knowing the difference helps you fix it. To diagnose: try the same prompt in a different tool or the raw API. If the problem persists across tools, it’s likely a model limitation. If it only happens in one tool, check how that tool injects context.

Trying to measure productivity improvement without baselines: You can’t prove AI made your team faster if you weren’t measuring before. Worse, once estimates become targets for measuring AI impact, developers adjust their estimates (consciously or not). Skip the productivity theatre. Instead, measure what matters: features shipped, customer value delivered, time from idea to production, team satisfaction.

What’s Next

This skill map is a snapshot. The tools evolve weekly. New capabilities emerge monthly.

If you’re on this journey, I’d like to hear what’s working for you. What skills have I missed? What resources have you found valuable?

Coming up: Putting these skills into practice. I’ll walk through setting up intelligent Engineering on a real project, covering tool configuration, context files, and workflow patterns that work.

If you’ve used Claude Sonnet in Claude Code, Cursor, Copilot, and Windsurf, you’ve noticed this. The model is identical, but the behavior varies. This isn’t magic. It’s context engineering.

In intelligent Engineering: Principles for Building With AI, I mentioned that “context is everything” and that “context engineering matters more than prompt engineering.” But I didn’t explain what that means or how to do it. This post fills that gap.

The Whiteboard

Imagine you’re in a day-long strategy meeting. There’s one whiteboard in the room. That’s all the shared space you have.

Your teammate is brilliant. They can see everything on the board and reason about it. But here’s the thing: they have no memory outside this whiteboard. What’s written is all they know. Erase something, and it’s gone.

Before the meeting started, someone wrote ground rules at the top: “Focus on Q1 priorities. Be specific. No tangents.” This section doesn’t get erased. It frames everything that follows. (That’s the system prompt.)

The meeting begins. You add notes, diagrams, decisions. The board fills up. You need to add something new, but there’s no space. What do you erase? The detailed debate from 9am, or the decision it produced? You keep the decision, erase the discussion. (That’s compaction.)

Three hours in, you notice something odd. Your teammate keeps referencing the top and bottom of the board, but seems to miss what’s in the middle. Important context from 10:30am is right there, but they’re not looking at it. The middle of the board gets less attention.

Someone raises a topic that needs last quarter’s data. Do you copy the entire Q4 report onto the board? No. You flip open your notebook, find the one relevant chart, add it to the board, discuss it, then erase it when you move on. (That’s just-in-time retrieval.) The notebook stays on the table. You reference it when needed, but it doesn’t consume board space.

By afternoon, old notes are causing problems. A 9am assumption turned out to be wrong, but it’s still on the board. Your teammate keeps building on it. The board is poisoned with outdated information. You need to actively clean it up.

There’s too much on the board now. Some notes are written in shorthand. Others are cramped into corners with tiny handwriting. Your teammate can technically see it all, but finding anything takes effort. Attention is diluted. (That’s context distraction.)

For a complex sub-problem, you send two people to side rooms with fresh whiteboards. They work independently, then return with one-page summaries. You add the summaries to your board and integrate the findings. You never needed their full whiteboards. (That’s sub-agents.)

The whiteboard is your teammate’s entire context window. What’s on it is all they can work with. Your job is to curate what goes on the board so they can focus on what matters.

What This Means Technically

The whiteboard story maps directly to how AI models process information.

System Prompts vs User Prompts

The ground rules at the top of the board are the system prompt. You didn’t write them. They were there when you walked in, set by whoever built the tool. They define how the model behaves, what it prioritizes, what it can do.

What you add during the meeting is the user prompt. Your requests, your context, your questions. It works within the frame the system prompt establishes.

The model sees both. But system prompts carry more weight because they come first and set expectations.

The Context Window

The whiteboard’s physical dimensions are the context window. There’s a fixed amount of space. Everything competes for it: system instructions, conversation history, files you’ve pulled in, tool definitions, and the model’s own output. When it fills up, something has to go.

Lost in the Middle

Remember how your teammate focused on the top and bottom of the board but missed the middle? That’s a real phenomenon. Research shows a U-shaped attention curve: information at the start and end of context gets more attention than information in the middle.

This means:

• Cramming everything into context can hurt performance
• Position matters: put important information first or last
• As context grows, accuracy often decreases

In Patterns for AI-assisted Software Development, I described LLMs as “teammates with anterograde amnesia.” They can hold information, but only within the context window. Understanding how to manage that window is key.

The Attention Budget

Even with everything visible on the board, your teammate can only actively focus on so much while reasoning. Each item costs attention. Add more, and something else gets less focus. Think of it as a budget: every token you add depletes some of the model’s capacity to focus on what matters.

How Different Tools Set Up the Room

Here’s why the same model behaves differently across tools: different rooms have different ground rules at the top of the board.

Take Claude Sonnet 4.5. Same teammate. But put them in different rooms:

Your teammate reads the top of the board and behaves accordingly. That’s why the same model feels different in each tool. The system prompt shapes everything.

This also explains why prompts don’t transfer directly between tools. A prompt that works well in Claude Code might fail in Cursor because the framing is different.

What Goes Wrong

When context fails, it fails in predictable ways. Recognizing these patterns helps you diagnose problems.

Context Poisoning

Early errors compound. Your teammate builds on incorrect assumptions, reinforcing mistakes with each exchange. By the time you notice, the board is thoroughly polluted with wrong information.

Fix: Use backtrack to undo recent turns. Claude Code, Cursor, and Windsurf all support this. If the pollution runs deeper, compact to summarize past the bad section. Clear is the nuclear option when context is unsalvageable.

Context Distraction

Too much information competes for attention. The model can technically process it all, but signal gets lost in noise.

On the whiteboard: shorthand, tiny writing, notes crammed into corners. Your teammate can see it all, but finding anything takes effort.

Fix: Keep context lean. Compact proactively. Don’t dump everything onto the board.

Context Confusion

Mixed content types muddle the model’s understanding. Code snippets, prose explanations, JSON configs, and error logs all blur together. The model can’t distinguish what’s an instruction versus an example versus context.

On the whiteboard: sticky notes, diagrams, tables, arrows, different colored markers. Your teammate can’t parse what type of information to use for what purpose.

Fix: Use focused tools. Don’t overload the board with too many formats or capabilities.

Context Clash

Contradictory instructions coexist. “Prioritize speed” in one corner. “Prioritize quality” in another. Your teammate sees both, doesn’t know which to follow, and produces something incoherent.

Fix: Keep instructions centralized and current. Review your context files periodically for contradictions.

Managing Context Well

Five techniques make a difference.

Just-in-Time Retrieval

Don’t paste your whole codebase onto the board. Reference specific files and let the tool search.

Bad: “Here’s my entire src/ directory. Now fix the bug.” Good: “There’s a bug in the date parser. Check src/utils/dates.ts.”

The notebook stays on the table. You flip it open when needed, find the relevant page, add it to the discussion, then move on.

Compaction

Context fills up during long sessions. Compaction summarizes conversation history, preserving key decisions while discarding noise.

When to compact:

• After completing a major task (before starting the next one)
• During long sessions when you notice drift
• Before context hits limits (proactively, not reactively)

You can provide custom instructions when compacting: “focus on architectural decisions” or “preserve the error messages we encountered.” This guides what gets kept versus summarized away.

My preference hierarchy:

1. Small tasks with /clear - fresh context beats compressed context
2. Early compaction with custom instructions - you control what matters
3. Early compaction with default prompt - still gives thinking room
4. Late compaction - avoid this

Late compaction (waiting until 95% capacity) is the worst option. The model has no thinking room, and the automatic summarization is opaque. You lose nuance without knowing what disappeared. Early compaction, ideally with custom instructions, gives you control and leaves space for the model to reason. Steve Kinney’s guide to Claude Code compaction covers the mechanics well.

Structured Note-Taking

For complex, multi-hour work, maintain notes outside the conversation:

• A NOTES.md file tracking progress
• Decision logs capturing why you chose specific approaches
• TODO lists that persist across compactions

The model can reference these files when needed, but they’re not consuming context constantly. The notebook on the table, not copied onto the board.

Sub-Agents

For large tasks, send people to side rooms with fresh whiteboards:

• Main agent coordinates the overall task
• Sub-agents handle specific, focused work with clean context
• Sub-agents return condensed summaries
• Main agent integrates results without carrying full sub-task context

This mirrors how teams work: delegate, get summaries, integrate. Claude Code supports this pattern for parallel issue work using git worktrees.

Tool-Specific Tips

Each tool has different mechanisms for managing what goes on the board.

Claude Code:

• CLAUDE.md files load automatically at session start. Keep them focused and current.
• Hierarchical loading: user-level, project-level, directory-level. More specific overrides more general.
• Trust the tool’s search. Don’t paste file contents manually unless retrieval fails.
• Use /compact between logical units of work.

Cursor:

• Rules files inject instructions with different scopes: global, project, file-type specific.
• Use @-mentions deliberately. More files isn’t better; relevant files are better.
• Keep rule files short. They add to every interaction.

Copilot:

• Lighter touch. Works best for autocomplete and quick suggestions.
• Less configurable context, so prompt quality matters more.

Windsurf:

• Memories persist across sessions automatically.
• Good for maintaining preferences and patterns over time.

Aider, Cline, and similar terminal-based tools follow the same principles. Different mechanisms, same underlying constraints. For a deeper comparison, see How to choose your coding assistants.

The Core Principle

Anthropic’s engineering team puts it well in their guide to context engineering:

Find the smallest set of high-signal tokens that maximize the likelihood of your desired outcome.

More context isn’t better. Relevant context is better. Your job is to curate what goes on the board so your teammate can focus on what matters.

Context drives quality. But “quality context” doesn’t mean volume. It means signal: information the model needs to reason correctly. Everything else dilutes attention.

What’s Next

Context engineering is a skill that develops with practice. Start by noticing when your tools perform well and when they drift. Ask why. Usually, the answer is in the context.

Take a few minutes to examine how your tool handles context. Where do instructions go? How do files get included? What happens during long sessions?

Understanding this is the difference between fighting your tools and working with them.

Coming up: Context engineering is one piece of the puzzle. In intelligent Engineering: A Skill Map for Learning AI-Assisted Development, I map out the full landscape of skills worth building.

I’ve spent the last two years applying AI across prototyping, internal tools, production systems, and team workflows. I’ve watched it produce an elegant solution in seconds, then confidently generate code calling APIs that don’t exist. I’ve seen it save hours on boilerplate and cost hours debugging hallucinated dependencies.

One thing has become clear: AI doesn’t make engineering easier. It shifts where the hard parts are.

The teams I’ve seen succeed with AI aren’t the ones using it everywhere. They’re the ones using it deliberately, knowing when to trust it, when to verify, and when to ignore it entirely.

Here’s a working set of principles I’ve found useful. They aren’t finished and will evolve with the tools. But they help keep me grounded in what actually matters.

intelligent Engineering Principles

These principles fall into two buckets: what is new, and what remains timeless but more important than ever.

AI-Native Principles

These principles exist because of AI. They address challenges that didn’t matter before.

AI augments, humans stay accountable.

AI can help you move faster and see options you’d miss on your own. But it can’t own the outcome. Engineering judgment stays with you. When something breaks in production, “the AI suggested it” isn’t an acceptable answer.

Context is everything.

AI output reflects what you put in. Vague requests get vague results. Bring useful context: project constraints, coding standards, relevant examples, what you’ve already tried.

As systems grow, context management becomes a discipline of its own. How do new teammates get AI tools primed with the right information? How do you keep that context current? When context exceeds what fits in a prompt, you’ll need solutions like modular documentation.

Smarter AI needs smarter guardrails.

Faster generation demands sharper review. AI-produced code still needs validation: Is it correct? Secure? Does it solve the right problem?

Shape AI deliberately.

I’ve seen teams adopt whatever AI tools are trending without asking whether they fit. Six months later, half the codebase assumed Copilot’s import ordering, onboarding docs referenced prompts that no longer worked, and no one remembered why. Decide upfront: where does AI help us? Where does it not? What happens when we switch tools?

Learning never stops.

At the start of 2025, AI practices evolved weekly. By year’s end, monthly. That’s still faster than most teams are used to. What didn’t work three months ago might work now. The only way to know is to keep experimenting.

I’ve settled on 90% getting work done, 10% experimenting. Try new ways to solve the same problem. Revisit old problems to see if there’s a simpler solution now. Check if techniques you learned last quarter still make sense.

Timeless Foundations

These aren’t new, but AI makes them more important.

Learn fast, adapt continuously.

Start small, validate often, and shorten feedback loops. If an AI-assisted workflow isn’t helping, change it. Don’t let sunk cost keep you on a bad path.

Fast doesn’t mean good.

AI makes it easy to generate code fast. That doesn’t mean the code is worth keeping. Unmaintainable, insecure, or rigid solutions cost more than they save. Build the right thing, not just the quick thing.

What This Looks Like in Practice

Here’s what this means day-to-day:

• I use AI to draft implementations, then spend more time reviewing than I saved generating. The review is where the real work happens.
• When AI suggests an approach, I ask “why?” If I can’t explain the choice to a teammate, I don’t use it.
• I’ve learned to be specific. “Write a function to parse dates” gets garbage. “Write unit tests for an ISO 8601 date parser, including edge cases for timezone offsets, leap seconds, and malformed input—then implement a function that passes them” gets something I can actually trust.
• I treat AI output like code from a confident junior developer: often correct, sometimes subtly wrong, occasionally completely off base.

The craft hasn’t changed. I still need to understand the problem, reason about edge cases, and take responsibility for what ships.

Skills Worth Building

Principles guide decisions. Skills make them possible.

Here’s what I’ve found worth investing in:

Context engineering matters more than prompt engineering. A clever prompt won’t fix bad context. I spend more time curating what information the model sees than crafting how I ask for things. Project documentation, coding standards, relevant examples. These matter more than prompt tricks.

Understanding tokens and context windows helps. You don’t need to become an ML engineer. But it helps to know why your 50-file codebase overwhelms the model, or why it “forgets” earlier instructions.

Agentic workflow primitives matter more than AI theory. You won’t build RAG systems from scratch. You’ll use tools with these built in. What matters is configuring them: hooks that customize behavior, skills that extend capabilities, context management that keeps information relevant. I spend more time learning how my tools’ hooks work or how to structure context files than reading ML papers.

For a comprehensive guide to building these skills, see A Skill Map for Learning AI-Assisted Development.

Why This Matters

I’ve seen what happens when teams adopt AI without thinking it through. Prototypes that demo well but collapse under real load. Codebases where no one understands why decisions were made because “the AI suggested it.” Bugs that take days to track down because the generated code looked plausible but handled edge cases incorrectly.

The failure mode isn’t dramatic. It’s slow erosion: teams that gradually stop reasoning deeply because the model provides answers quickly.

The alternative isn’t avoiding AI. It’s using it with intention. The engineers I’ve seen do this well have gotten faster and more thoughtful. They use AI to handle the routine and focus on the hard problems.

What’s Next

These principles aren’t final. I expect to revise them as tools improve and as I learn what actually works versus what sounds good in theory.

If you’re experimenting with AI in your engineering work, I’d be curious to hear what’s working for you. What would you add? What would you challenge?

Credits

This blog would not have been possible without the review and feedback from Greg Reiser, George Song and Karthika Vijayan for reviewing multiple versions of this post and providing patient feedback 😀.

This content has been written on the shoulders of giants (at and outside Sahaj).

I have talked about teams needing to have a world class developer experience as a pre-requisite for a well functioning team. When teams lack such a setup, the most common response is a lack of time or buy in from stakeholders to build these things. With AI coding assistants being readily available to most developers today, the engineering effort and the cost investment for the business lesser reducing the barrier to entry.

Current State

This post showcases an actual codebase that has not been actively maintained for over 5 years but runs a product that is actively used. It is business critical but did not have the necessary safety nets in place. Let us go through the journey, prompts inclusive, on how to make the code quality of this repository better, one prompt at a time.

The project is a Django backend application that exposes APIs. We start off with a quick overview of the code and notice that there are tests and some documentation but a lack of consistent way to run and test the application.

The Journey

I am assuming you are running these commands using Claude Code (with Claude Sonnet 4 in most cases). This is equally applicable across any coding assistant. Results will vary based on your choices of models, prompts and the codebase.

Setting up Basic Documentation and Some Automation

If you are using a tool like Claude Code, run /init in your repository and you will get a significant part of this documentation.

Can you analyse the code and write up documentation in README.md that
 clearly summarises how to setup, run, test and lint the application.
Please make sure the file is concise and does not repeat itself. 
Write it like technical documentation. Short and sweet.

Next step is to start setting up some automation (like just files) to help make the project easier to use. This will take a couple of attempts to get right but here is a prompt you can start off with

Please write up a just file. I would like the following commands
`just setup` - set up all the dependencies of the project
`just run` - start up the applications including any dependencies
`just test` - run all tests
If you require clarifications, please ask questions. 
Think hard about what other requirements I need to fulfill. 
Be critical and question everything. 
Do not make code changes till you are clear on what needs to be done.

This will give you a base structure for you to modify quickly and get up and running. If you README.md has a preferred way to run the application (locally vs docker), the just will automatically use it. If not, you will have to provide clarification.

Setting up pre-commit for Early Feedback

Let’s start small and build on it.

Please setup pre-commit with a single task to run all tests on every push.
Update the just script to ensure pre-commit hooks are installed locally
 during the setup process.

We probably didn’t need to be this explicit but I find managing context and keeping tasks small mean I move a lot quicker.

Curating Code Quality Tools

Lets begin by finding good tools to use, create a plan for the change and then execute the plan. Start off by moving Claude Code to Plan mode (shift+tab twice)

What's a good tool to check the complexity of the python code this
 repository has and lint on it to provide the team feedback as a 
 pre-commit hook?

It came back with a set of tools I liked but it assumed that the commit will immediately go green. In an existing large codebase with tech debt, this will not happen. Let’s break this down further.

The list of tools you're suggesting sound good. 
The codebase currently will have a very large number of violations. 
I want the ability to incrementally improve things with every commit. 
How do we achieve this?

Creating a Plan

After you iterate on the previous prompt with the agent, you will get a plan that you’ll be happy with. The AI assistant will ask for permission to move forward and execute the plan but before doing so, it will be worth creating a save state. Imagine this as a video game save, if something goes wrong, come back and restore from this point. This also allows you to clear context since everything is dumped to markdown files on disk.

Can you create a plan that is executable in steps?
Write that plan to `docs/code-quality-improvements`.
Try to use multiple background agents if it helps speed up this process.

Give it a few minutes to analyse the code. In my case, the following files were created. README.md says that “Tasks within the same phase can be executed in parallel by multiple Claude Code assistants, as long as prerequisites are met”. You are ready to hit /clear and clear out the context window.

Phase 1 sets up the basic tools, phase 2 configures them, phase 3 focuses on integration and automation and phase 4 adds monitoring and focuses on improving the code quality.

Before executing the plan, I commit the plan (docs/code-quality-improvement). This allows me to track any changes that have been made. When executing the plan, I do not check in the changes made to the plan. This allows me to drop the plan at the end of the process. As a team, we have discussed potentially keeping the plan around as an artifact. To do so, you would have to ask Claude Code to use relative paths (it uses absolute paths when asking for files to be updated in the plan).

Executing the Plan

I would like to improve code quality and I have come up with a plan to do 
so under `docs/code-quality-improvement`.
Can you analyse the plan and start executing it? The `README.md` has a 
quick start section which tasks about how to execute different phases of the 
plan. As you execute the plan, mark tasks as done to track state.

You will notice that Claude Code will add dependencies to requirements-dev.txt and try to run things without installing them. Also, it will add dependencies that do not exist. Stop the execution (by pressing Esc ) and use the following prompt to course correct

For every pip dependency you add to `requirements-dev.txt`, please run 
`pip install`. 
Before adding a dependency to the dependency file, please check if it is 
available on `pip`.

Once phase 1 and phase 2 of the plan are complete, the following files are created and ready to be committed.

When the quality gates are added on phase 3, run the command once to test if everything works and create another commit. After this, I had to prompt it once more to integrate the lint steps into a simplified developer experience.

Please add `just lint` as a command to run all quality checks

Test the brand new lint command and then run a commit. Ask claude code to proceed to phase 4.

You might see Claude Code doubt a plan that it has created. It is a good question because the system is functional but if we prefer the more advanced checks, we should request it pushes on with Phase 4 implementation.

After phase 4, we have a codebase that checks for code quality every time a developer is pushing code. Our repository has pre-commit hooks for linting, runs all quality checks once before pushing. The quality checks will fail if the code added has unformatted files, imports in the wrong order, flake8 lint issues or functions with higher code complexity. It checks this only in the files being touched (because we told it that we had debt that needs to be reduced and all checks will not pass by default)

You still have debt, lets go over fixing this in the next step.

Fixing Existing Debt

Tools like isort can highlight issues and fix them. You should start off running such commands to fix the code. On most codebases, this will touch almost all of the files. The challenge with this is that all the issues that cannot be fixed automatically (like wildcard imports) will need to be fixed manually. This is where you make a choice either to fix issues manually or automatically. If you’re using Claude Code to fix these issues and there is a large number, you’re probably going to pay in upwards of $10 for this session on any decent sized codebase. I recommend moving to GitHub Copilot’s agent to help push down costs here.

Ask your coding assistant of choice to run the lint command and fix the issues. Most of them will stop after 1–2 attempts because the list is large. You can tell it to “keep doing this task till there are no linting errors left. DO NOT stop till the lint command passes”. If your context file (CLAUDE.md) does not talk about how to lint, be explicit and tell your coding assistant what the command to be run is.

What is Left?

If you look at the gradual-tightening task, it created a command to analyse the code and keep being gradually more strict. This command can either be run manually or automatically on a pipeline. One of the parameters it changes is the max-complexity which is set to 20 by default. This complexity will be reduced over a period of time. Similarly, the complexity check tasks have a lower bar to begin with and should be improved periodically to tighten the quality guidelines on this repository.

While our AI coding pair has helped design and improve the code quality to a large extent, the last mile has to be walked by all of our teammates. We now have a strong feedback mechanism for bad code that will fail the pipeline and stop code from being committed or pushed. The last bit requires team culture to be built. On one of my teams, we had a soft check in every retro to see if every member had made the codebase a little bit better in a sprint. A sprint is 10 days and “a little bit” can include refactoring a tiny 2–3 line function and making it better. The bar is really low but the social pressure of wanting to make things better motivated all of us to drive positive change.

Having a high quality codebase with a good developer experience is not a pipe dream and making it a reality is easier than ever with AI coding assistants like Claude Code or Copilot. What have you been able to improve recently? 😃

Coding assistants like Cursor, Windsurf, Claude Code, Gemini CLI, Codex, Aider, OpenCode, JetBrains AI etc. have been making the news for the last few months. Yet, the choice of tools is a lot harder and limited for some of us than it seems.

TL;DR: OpenCode > Claude Code > Aider > Copilot > *

Understanding the tools

Not all tools are created equal. Tools evolve fairly rapidly so the examples listed here might be invalid fairly soon.

You can plot the different types of coding assistants on a graph showcasing the amount of human involvement required (lesser involvement = more automation). The first GitHub Copilot release I used allowed tab completions. It would either complete single lines or entire blocks of code. You could describe your intent by creating a function with a good name or by writing a comment. GitHub Copilot then supported inline prompting or chat sessions.

Coding agents are the current state of the art toolset for most developers on a day to day basis. They allow you to have conversations with them and you should treat them as team mates, albeit ones with anterograde amnesia.

Some problems can be parallelised and background agents triggered locally are incredibly powerful. Claude code supports subagents is frequently used for analysis and solving multiple issues in parallel using git worktrees. Similarly, some people hook up agents to remote instances for things like code reviews using Claude code or Copilot.

The extreme version of this is pure vibe coding. There is enough content out there about why this is a bad idea and the number of issues on real systems because of this.

Challenges with using these tools

When picking up a tool, I have started looking at different aspects of these tools

Choice of models

LLMs change quite quickly. Claude Sonnet 3.7 started off being the favourite model for most developers I know. When Claude Sonnet 4 came out at the same cost as 3.7, it became the new favourite model. Claude Opus 4 is great for larger codebases but expensive.

As I write this (mid-July 2025), the word on the street is that Grok 4 is currently the best model on the block. Choose something that has good coding insights and a large context window. Claude Sonnet has some of the smaller context windows but is tuned quite well for software development.

Cursor supports most of the best models and provides diversity. Tools like Claude Code and Gemini CLI are built and maintained primarily for use with a single model.

Ease of use

This one is fairly subjective and dependent on the developer’s preference. Tools like Cursor are VS Code forks and thus provide tight integration with the editor. Others like Claude Code, Codex and Gemini CLI run on the terminal. Claude Code provides decent integration with the IDEs from the JetBrains family and thus provide good support to pair with your AI assistant.

Speed factors into ease of use too. While Jetbrains AI is the best integrated tool amongst all of these (if you prefer using their IDEs), their AI tool is one of the slowest. Slower tools mean slower feedback cycles. Slower feedback cycles are some of the worst things for dev experience.

Cost per change

Cost pays a huge part in someone’s choice of tools and running LLMs are fairly expensive to run. Most tools charge you per use, some by tokens, some by APIs. Since we’re in the relatively early days of these tools and they are competing to capture the market, some still provide fixed investment offers in exchange for “unlimited plans.

Cursor used to be $20/month with unlimited usage till June 2025. While all “unlimited” usage is rate limited, if the usage limits are generous or the rate limits are not severe, users can manage to have a decent developer experience. More recently, Cursor updated their prices to make the $20/month Pro plan for “light users”. Daily users are recommended to use their $60/month Pro+ plan and power users are recommended to use their $200/month Ultra plan. Users on reddit have complained about how the Ultra plan is insufficient, though Cursor’s documentation says that it should be sufficient. This seems to primarily be because of heavy Claude Opus 4 usage, one of the most expensive models.

Another fixed usage tool is Claude Code for individuals with it’s Pro and Max plans. The $100/month Max plan seems to be the sweet spot for most heavy users and is probably the best value for money, at least until you look at the licensing.

Google’s Gemini CLI, at launched, announced the most insane free tier (that allows you to spend an estimated $620/day) but at the cost of training on your projects. More on this, in the next section. The free tier might not be this generous forever so if the “training on your data” bit isn’t a concern, enjoy Google’s generosity.

IP ownership indemnity and licensing

Licensing is a complicated topic and I go off of the advice that people much more qualified than me give in this space. The current understanding of this space is that you want to be on

1. company licensing (avoid individual licenses)
2. a tool that does not train on your data
3. provides you indemnity against IP claims

You should avoid individual licenses since the protections usually apply to you, not the organisation you work for. If you work with a services company and create IP for your clients, you want to avoid the risk of the protections not covering your clients.

Avoid tools that train on your data if you’re building something commercially. If you’re on a FOSS tool/system, you can ignore this fact. Google Gemini CLI’s free tier is a great example of this. They get to use your data to make the system better in exchange for you having a good coding assistant free of cost.

Anthropic, the creator of Claude Code, indemnifies its commercial users against lawsuits. Most other tools tend to do this too. Interestingly, Cursor does not, at least as of the writing of this article. Their MSA provides this protection, however, they only do this for customers signing up for more than 250 seats. This may change in the future and talking to their support is the best way to clarify this.

What do I use and recommend at this point?

For team members who are new to using coding assistants, start off with Copilot where users will appreciate the fixed cost. Learn, experiment. Strengthen your core skills in this new world: Prompt Engineering and Context Engineering (more on these skills in another blog).

When you have mastered these skills, you should consider moving to an API based tool that allows you to switch between models. Personally, I’m a fan of the Claude Sonnet and Opus models over OpenAI (and to some extent, Gemini). If you can manage costs well, move to Claude Code (or an open source tool like OpenCode or Aider). I would put OpenCode above Claude Code due to it’s flexibility.

In the last post — AI for Software Engineering, not (only) Code Generation — we explored how AI is transforming software engineering beyond just writing code. Now, let’s look at what that means for teams and individuals in practice.

There are a few patterns that people running teams and on teams that are going to build software with assistance from AI tools should remember.

For people building teams

Focus on value

With the AI ecosystem shifting weekly, C-level and VP-level stakeholders who prioritise modular documentation, model pairing, scoped context, and tooling agility will drive the highest ROI while keeping teams nimble and ready for whatever comes next. Make it work, make it right and then make it fast/cheap.

Journey per software delivery stage, one stage at a time per team

This journey is going to be transformational for teams. Like most transformations, you do not want to change too much too quickly.

When bringing change to a single team, introduce it one software delivery stage at a time to easily verify effectiveness. In a large organisation, you could try different tools for the same stage on different teams to A/B test effectiveness while taking into account the nuances of the individual teams themselves. We don’t recommend this approach if you would like to converge towards a single tool throughout the organisation because changing tool choices after the team gets used to it causes more friction.

When you have multiple teams willing to take this journey, you can have each of them pick tools in different stages to help reduce the time that your organisation takes to make a decision on a toolset. A couple of teams can try AI tools for requirements analysis while others can try agentic coding tools for development.

Expect a learning curve

Especially if you’re an experienced developer, you will feel slower when you start off on this journey. This is no different than working with a new teammate and feeling that your overall productivity is lower. You trade off your own speed against the value you will get when your teammate is onboarded and can deliver by themselves.

From our experience, you are looking at a 2–4 week drop in perceived productivity before the gains will start showing up. As a result, the costs will go up (slower delivery and cost of tools) before they come back down (faster delivery and more time to focus on quality).

Quality guardrails are a prerequisite

Do not bolt on quality and security guardrails after the fact. Start with them. Ensure a robust test pyramid and implement shift-left strategies for both testing and security, enabling quick and early feedback. These guardrails will be invaluable when your team is moving at breakneck speeds through newer features.

If you don’t have these guardrails first, you can use AI to help generate them and review these plans. Like the Maker-Checker process, if an AI coding assistant has helped you plan and create these guardrails, they should be thoroughly reviewed by someone who has the expertise in these fields to catch the small bugs that can have disastrous consequences later.

Autonomous agents are far away

Humans are required in the loop for software development. 10+ years after the first demos of driverless cars, we’re still waiting for a general purpose implementation. While we have made massive progress, it takes time. While agents have made massive progress in the last 2 years, we still need to exist to make sure things work well and that the systems are maintainable. The skill to build maintainable systems is more important now than ever.

Watch out for ‘AI Slop’

Without the right guardrails and structures in place, teams will produce more code, faster while sacrificing quality and security. Teams that have been given access to AI tools without helping them build skills first often point out longer pull requests coming in faster than ever before making people reviewing the code a bottleneck. Eventually, the reviewers end up accepting pull requests due to pressure or fatigue leading to important issues being missed.

Individuals should focus on small chunks of work and teams should look at key metrics to measure the effectiveness of their tool usage (we talk about both of these later in the post).

Changes to individual responsibilities and team composition over time

If teams in your organisation currently contain distinct individuals playing different roles like business analyst, architect, developer, quality analyst, infrastructure engineer and production support engineer, you will see that the distinct responsibilities of these roles will rely less on administrative tasks freeing each of them to focus on thinking strategically and the core responsibilities of their roles. Different organisations will see a merger of different roles. Some will see a merger of the business analyst and product manager roles. Some will see product and project managers merge. Some will see project managers’ responsibilities be split between technical leads and product owners.

In doing so, individuals will emerge that pick up or demonstrate their ability wear multiple hats for example, talk to the business, design the system, develop, validate, deploy and monitor it. These individuals will understand the challenges of the business and work end to end to address it. We have been calling such individuals Solution Consultants at Sahaj and believe that most teams will need such individuals on their team in the near future once they leverage AI in their delivery.

Beware of reduced intuition for decision making

As teams move towards using automated notetakers to help capture more detailed conversations, we should be on the lookout for a few anti-patterns

While conversation summaries help with a quick read, they are often misleading or inaccurate. Please read the full transcription to help improve confidence in what was spoken about. Transcripts are not a replacement for actually having real conversations, an anti-pattern we have seen come up on recent teams.

Transcripts are also not a replacement for remembering context yourself. Context helps build intuition for decisions and one of our worries is that intuition will reduce over a period of time.

For people on teams

The ‘new teammate’ mindset

Treat the AI system as a new team mate or a collaborative partner and not a tool. You can use a tool, be unhappy about the way the tool works and stop using it. When a new team mate joins your team, the fundamental thought process is different. You try to onboard the team mate and give it better context. Writing good instructions or prompts is key to success.

LLMs are like team mates with anterograde amnesia. They can have some memories but these are fairly limited by the size of their context windows. Understanding how to manage context windows is key to being able to work with our new team mates effectively. Keep only what is necessary in the context window and clear it when it isn’t required. Common context should be added to a file (check rules section below) and included only when necessary.

If your prompts to a coding assistant are vague, the tool will keep going around in circles and not make any progress on the task or do the wrong thing.

For example, when you ask the agent: I have noticed that [http://localhost:4000/create-profile](http://localhost:4000/create-profile) has alignment issues and contains text that is spreading outside the buttons. Can you please fix this?

If the agent has access to the puppeteer MCP, it will open up the UI, take a screenshot, process and fix it. If your application has a login page, it will see that the Create Profile view is not being loaded and decide to “fix” this issue by removing authentication 😞. Adding “Please wait for me to login if required” to the prompt helps avoid this issue.

If your prompts have not told the system that you need a solution that has been simplified or one that does not hard code solutions, it will not follow these instructions. Add your general coding standards to a document and include that in the base context. If you have rules around test quality, split that into a smaller document explaining what good tests look like for the team.

Small chunks of work

Break your work down. Reviewing a 1000 line review has always been hard. You can generate large code diffs with AI quickly. You, the developer, are the bottleneck. You are still responsible for quality and security.

Work on smaller chunks. Review regularly. Do small commits. Age old practices still apply.

Configure the tool based on your team’s rules

Each tool requires configuration. Configurations take time to test. It might take a few tries over multiple days to get these configurations correct. Each tool has a different way to be configured and there is no standardisation. In the Agentic code pairing tool space, every tool has its own configuration mechanism. Cursor has Cursor Rules, Claude has memory, Windsurf has Memories & Rules and IntelliJ’s Junie has guidelines. Each of these looks like a markdown file but has slightly different formats. If you’re experimenting between multiple tools (or different teammates prefer different tools), you will have to keep these rules in sync by hand. What’s worse is that the same instructions do not have the same effectiveness across different tools because their system prompts are different. Testing regularly and tweaking is key. Tools also rapidly update. Claude Code releases every couple of days (at the time of writing). Rules may need to be updated based on changes to the tool of your choice.

Shift in time spent on different responsibilities

Teams will increasingly spend more time upfront in planning what needs to be built and what the right thing to build is than in actually building things. This does not mean that teams are walking away from agile but truly embracing it. The time spent on analysis and planning will go up as a proportion but the overall time taken to deliver a version will go down. Each of the individual activities (analysis, development etc.) will be done in thin slices helping build the system up incrementally.

Over-reliance on AI instead of thinking and remembering yourself

Since AI works fast, it’s easier to be lulled into a sense of security and thus have a sense of reliance on the tools. Over time, some individuals may spend less time thinking critically and making decisions.

For example, if a good note-taking app takes notes and summarises them correctly 95% of the time, it is easy to forget that the 5% of mistakes, especially if they happen in critical parts of the conversation, can be quite expensive to fix. Summaries are good but they are not a replacement for reading the transcript which itself cannot beat actually having a conversation with people.

We need to use these systems to help us be better at our roles. Critical thinking is not optional, now more so than ever. We need to put guardrails in place to spot and correct intellectual laziness. If an issue is found that you missed during review, check if you thought about it critically enough. Do so for teammates too and help provide feedback if they are slipping.

How do you know AI is helping software delivery?

Use both qualitative and quantitative measures. Early stages focus on “leading” indicators: developer sentiment, tool usage, and workflow metrics. Conduct developer surveys and track AI usage statistics (active users, acceptance rates) as GitHub recommends. Complement these with engineering metrics: cycle time (time from commit to deploy), pull-request size and review duration, deployment frequency, and change‑failure rates. These DORA‑style metrics help ensure speedups don’t sacrifice quality. Align these KPIs to business outcomes (e.g. shorter time-to-market, fewer critical bugs). Set “clear, measurable goals” for AI use and monitor both productivity and code quality over time.

Up next, we’ll dive into strategies for managing tech debt and elevating developer experience in a world where AI is part of the team. We’ll explore why it’s now easier than ever to stay ahead of the curve — and share the exact prompts and techniques that make it possible.

Credits

This blog would not have been possible without the constant support and guidance from Greg Reiser, Priyank Gupta, Veda Kanala and Akshay Karle. I would also like George Song and Carmen Mardiros for reviewing multiple versions of this post and providing patient feedback 😀.

This content has been written on the shoulders of giants (at and outside Sahaj) that I have done my best to quote throughout.

Everyone has been talking about using coding assistants to aid with software delivery. There is more to delivering good software than writing code.

Every software development project requires a few different activities from analysis (what), to planning and design (how), to development (build), to testing (validate), to deployment (implement). Each of these activities depends on different skills and techniques that can benefit from the effective use of modern AI technologies.

All software development methodologies, from waterfall to the different agile techniques, fundamentally follow the same cycle. We feel this cycle is not changing yet but there are improvements waiting to be unlocked for organisations.

This post aims to demonstrate how teams of the future can gear themselves to build better products faster.

Use of AI tools across software delivery

The tools mentioned in this section are examples to help the reader understand the idea and not recommendations on what to use.

During Analysis

Improved analysis

Many teams have integrated AI into their analysis process. Starting with single agent flows that support definition of features, epic and stories, to multi-agent flows that help with addressing different parts of a problem space in parallel. My colleague Carmen Mardiros showcases how to revise a plan using Claude Code where individual agents perform specific tasks to help the analyst optimise a plan before execution. Effectively using AI in support of critical analysis and planning can provide benefits beyond basic requirements definition. Multi-agent systems out-perform single agent systems but spend significantly more tokens (and thus money) to do so.

Taskmaster is an AI powered tool that, together with an interactive coding assistant such as Claude Code, can serve as a virtual technical project manager by helping with defining requirements, offering feedback on edge cases, writing stories and setting up and managing the product backlog.

Since you can also ask Claude Code to analyse the codebase to identify technical debt, you can use the same tools to manage both the technical and feature backlogs of the product. This is particularly important when working with mature (legacy) systems as teams and product owners often struggle with balancing technical debt reduction (payback) and new feature development. Although these tools do not replace the expertise required to effectively manage a backlog and prioritise work, they can significantly reduce the administrative burden of doing so.

If all requirements are documented as PRDs, it becomes easier to measure drift as well as look at cards that might be created but might have parts that have already been implemented. You can run this analysis as a weekly or monthly job to clean up your backlog of tasks that are no longer needed.

Not all administrative tasks have been eliminated. When you transition from PRDs to epics on your backlog, there is a time period when both remain active and during this time, the two need to be consciously kept in sync. Over a period of time, the importance of the PRD wanes and it can be killed off. The same is true for other transitions like the one between stories and code.

Changes in roles for Business Analysts and Project Managers

The roles of business analysts included note taking, summarising and analysing and helping shape the right product for the business. This role is shifting to focus on being more strategic in nature focusing on finding good opportunities for your products, taking away the transcription/administration parts of the role. Similarly, the roles of Project Managers will include less time on administrative tasks and more time on making sure the right features are being built.

This is true for all roles we’re going to be speaking about in this post to some extent, calling this out explicitly since this is the first.

Improved iterative UI/UX design

Tools such as Canva and Figma have helped minimise the time taken to go through a complete feedback cycle with users. AI tools have now started linking up with these tools to help spot implementation drift during development. These tools also have the ability to spot requirements gaps and help us foresee problems. More on this during the feedback cycles section.

Clair Mary Sebastian also talks about using generative AI for requirements analysis and wireframing using OpenAI’s APIs alongside Figma’s wireframe designer.

AI note taking apps for requirement analysis

Copilot4Devops that will take text summaries and help generate user stories or feature specs. This can be a particularly powerful technique to aide in quicker iterations with generating stories and feature specs.

Note taking apps like fireflies.ai have fairly accurate notes across multiple languages with user detection in conversations and help improve user experience and recall for conversations.

While conversation summaries help with a quick read, they are often misleading or inaccurate. A best practice (or should we say “must have practice”) is for participants to review the notes shortly after the meeting and correct any errors before the notes are accepted. In addition to preventing the dissemination of inaccurate information, this practice improves information retention amongst participants and contributes to an improved shared understanding. This is in contrast to the anti-pattern of relying on unreviewed transcripts and meeting notes, an anti-pattern that discourages critical thinking and delays establishment of a shared understanding that is critical to successful delivery.

Transcripts are not a replacement for actually having real conversations, an anti-pattern we have seen come up on recent teams. Transcripts are also not a replacement for remembering context yourself. Context helps build intuition for decisions and one of our worries is that intuition will reduce over a period of time.

Improved communication and context

Currently, users from the business (or product owners as a proxy) work with business analysts from delivery teams to collaboratively help shape the product. This communication usually requires experienced product owners who understand technology well enough at a distance to know what questions to ask and how to shape the conversation to build quick consensus on what the product’s vision is. This communication also requires experienced business analysts who know how to extract details of how the system should work, anticipate challenges during building the product and pre-empt them with questions. Teams who do a good job at analysing the system require individuals at the top of their game. If either of these individuals does not have the pre-requisite knowledge, communication is sub-optimal.

We see that this status-quo is ripe for disruption. Doing so requires us to build a system (or product) that absorbs domain context before it can be used.

Since most teams are distributed, a conversational AI can help users prepare for their synchronous or asynchronous communication with the team given that the AI has the persona of a developer who is an expert at the specific tech that is used to work on the product. Similarly, delivery team members can use a conversational AI system to help understand the business context better and anticipate pushback and prep for it. Being able to understand the devil’s advocate stance in their head and prepare for it is something most people struggle with. Important conversations still happen through direct communication, however, both the users and the business analysts can help pair on preparing for the actual conversation with real people on the other side.

Over a period of time, the conversational AI system can help improve the quality of preparation conversations for both actors providing quicker feedback.

During System Design

AI makes it possible to more quickly and thoroughly define and compare different solution designs for a given problem space. The ability to quickly and thoroughly evaluate the impact of different architectural decisions can multiply the value of experienced architects and may even enable more advanced practices such as emergent architecture as AI can help teams safely adjust the solution design as requirements change or new requirements emerge.

When a system is built, the system design is built to meet some constraints and have a target state. Both the target state and constraints evolve over time. Good teams will track these constraints in the beginning and through the evolution of the product as ADRs and fitness functions. Some teams find it hard to keep track of the delta between the current and target state (current debt). Using AI tools, this debt is easier to identify, track and address. Teams can use specific prompts in different areas to identify these challenges and help evolve the system in the right direction.

Tools like eraser.io exist to allow generation of architectural documents through text. Combining this with the ability to generate documentation based on the code, systems can ensure architectural documents are always up to date.

During Development and validation

In today’s fast-evolving AI landscape, engineers must embrace a dual-mode workflow (planner and executor) to get the most out of coding assistants. As a planner, you leverage a high-reasoning model (for example, Claude Sonnet 4 over 3.7 or GPT-4o) to deconstruct monolithic docs into modular guides (e.g. splitting a bulky claude.md into coding-practices.md and development-workflow.md), map out architectural changes, and draft a detailed implementation roadmap. Once the blueprint is locked in, switch to a specialized coding model (like Sonnet, GitHub Copilot with tailored instructions, or Claude Code) for hands-on development, refactoring, and validation. By matching each task to the model best suited for it and scoping prompts to only the relevant files or services you streamline token usage, accelerate processing, and cut context-window bloat.

Executing at scale also demands a culture of experimentation and flexibility. Expect a learning curve as teams test different assistants (Copilot, Cursor, Claude-Code, etc.) and prompt strategies for different tasks like migrating an entire codebase versus tweaking a single method signature, for example. Build in continuous feedback loops around prompt-to-PR cycle times, code quality metrics, and token costs to identify what works best in each scenario. Agentic integrations via Model Context Protocols and tools like Puppeteer, Slack bots, and GitHub Actions can then automate routine tasks — from branch creation to dependency updates and test orchestration right within your existing toolchain.

During Deployment and Operationalisation

Over the past decade, practices in the DevOps space have changed quite significantly with the focus on automation (CI/CD) observability and improved monitoring tools. As this data became more centralised in platforms like AppDynamics, DataDog and NewRelic, these systems have been able to spot errors, intelligently alert users and help spot anomalies.

Platforms like Harness now support automated error analysis to help understand the root cause of issues and help provide steps to fix them.

During Feedback Cycles

Traditionally, individuals caught drifts in software development. There are tools being built in place to help catch different types of drift. Tools such as Cubyts catch both requirement drift (between requirement specs and stories) and implementation drift (between requirement specs, application mock ups and implementation). This is possible because these tools connect with tools like JIRA, Figma, GitHub etc. to analyse the contents of that platform and find possible challenges using the capabilities LLMs provide.

How do you enable this transformation

Preparation

1. Identify a candidate project
2. Ensure the candidate project has good safety nets
3. Ensure the candidate project has a stable product team with good shared context
4. Identify the right stage of software development, which is most painful and will benefit from introducing AI tools
5. Identify seed individuals with prior experience in the space, the right opinions and the ability to mentor team members
6. Identify the tool to introduce
7. Set up success criteria for this transformation

The journey

1. Set up time to up-skill team members (on the skills from the “For people on teams” section). Pair team members with seed individuals for maximum effectiveness.
2. Set up weekly retrospective meetings to catch trends and course correct as necessary. Timely feedback is critical.
3. Set up a checkpoint to see if the team members require less support from seed individuals weekly. Until a threshold of independence is reached, keep repeating steps 1–3.
4. Seed individuals depart from the team and only join retrospectives for support.
5. Set up a checkpoint to check if seed individuals are required in the retros and to confirm that the team is meeting the success criteria.

The 4-week period are indicative examples of what teams may need. Tweak the time period on a need basis.

AI’s role in software engineering goes far beyond code generation — it’s reshaping how we design systems, make decisions, and collaborate. To truly unlock its potential, we need to rethink not just our tools, but how our teams operate. In the next post, we’ll explore patterns for AI-assisted software delivery — focusing on how to build more effective teams, and how individuals can work differently to make the most of AI in their day-to-day practice.

Credits

This blog would not have been possible without the constant support and guidance from Greg Reiser, Priyank Gupta, Veda Kanala and Akshay Karle. I would also like Swapnil Sankla, George Song, Rhushikesh Apte and Carmen Mardiros for reviewing multiple versions of this document and providing patient feedback 😀.

This content has been written on the shoulders of giants (at and outside Sahaj) that I have done my best to quote throughout.

Developer experience (DevEx) isn’t just about fancy tools or slick UIs - it’s about removing friction so teams can move with confidence, speed, and clarity. In high-performing teams, great DevEx means fewer context switches, faster feedback loops, and more time spent actually building. In this post, we’ll explore the five non-negotiables every codebase should have to support world-class collaboration, and we’ll map out a practical DevEx stack to help your team deliver better products, faster.

The Five Non-negotiables

I. Project readme

Short, sweet and simple

Write a short note with a few lines on what this codebase is responsible for. Indicate the setup process and lifecycle to go to production. Code should act as documentation and anything that code will not document as obviously (what are the first set of things you should read) should be in here.

II. Automated setup

A single command to get your entire workstation setup.

I am a huge fan of using shell scripts for smaller projects and justfiles for larger ones. This isn’t about tools. This is about your experience.

Run just setup and have a workstation that is ready to go (including requiring node/python, installing all the dependencies and setting up a database, if required. I expect other obvious commands like just run, just lint, just test and just build. I admit that I have been spoiled by gradle and maven in JVM land and clearly have withdrawal symptoms in the land of the snakes.

Take this a step further and automate test data creation. If your application is stateful, please generate the test data on startup. This way, you are ready to test what you need the moment your application starts. Test data setup might add a few seconds to your startup but it will save you minutes in testing things and much more than that in your emotional happiness. If you are building an e-commerce website, create a few product categories, products in each of the categories and a few test users. Make sure your test user has elevated privileges to begin with making it easier for you to start testing things. The single just run command should have you ready to test your scenarios.

III. Iterate fast

Faster the feedback, the better

I like fast iterations. Left on my own, I’d commit every 5–10 minutes; sooner, if I can get away with it. This includes the time it takes me to lint and test. This means fast code linting and tests. I love code linting tools that take less than a second and unit tests that take less than 5 seconds across the entire project. If running all tests takes more than 5 seconds, I’ll run them before a push. If it takes more than a minute, I’m refactoring/optimising something.

There are enough engineering techniques to go fast. Got a large number of tests? Run them in parallel. Integration tests take time? Share container context and database test containers.

Once you get used to this, you will not go back.

IV. Enforced pre-commit/pre-push checks

Shift feedback leftward

Use frameworks like pre-commit (others exist for most toolchains) to run your entire CI safety net locally. Early feedback is key. Lint code with every commit. Linters should spot issues (like increased complexity, dead code, etc.) early and format code consistently. Test everything before pushing.

V. Everything runs locally

Nothing should require the internet or external resources if possible

Can you run your code locally? Sounds like a silly suggestion but I bet that there is at least one team still working with an old system that’s written in C that logs into a remote machine and writes code in vim without any setup for code completion, early compilation feedback or running code in an “IDE like setup” with world-class debugging support.

Use a proper IDE. I personally love the IntelliJ suite of tools for most languages. Some of my teammates are emacs and vim power users who have the same setups (code completion, auto-compilation, error detection, running code, and debugging support). IntelliJ even comes with its own set of profiling tools that are a real timesaver for me and easily worth the cost of usage.

A teammate once asked “If you were to get on a flight, could you continue to write code”. This was not a hypothetical question as we used to travel every week and spend 3+ hours on a flight, time you’d like to make good use of. Having a toolset that you can go offline and work comfortably in, even when travelling is a really nice experience as a developer.

The DevEx Stack

Want to go beyond the non negotiable items and dive deeper into improving your team’s DevEx? Here’s a stack with some techniques, tools and practices to try out.

This section is going to be heavy with crosslinks to other articles to keep this article short for people who already know some of these concepts.

        Layer                          Tools/Practices
Code Quality            Linters, Formatters, Typing, Modular Design
Automation              Pre-commit, CI/CD, Makefiles, Containerisation
Testing and Validation  Fast tests, Coverage, Contracts, Security Scans
Documentation           Onboarding, Readmes, ADRs, Comments, PR Templates
Culture and Workflow    Git hygiene, Blameless retros, Tech debt tracking

Foundational code practices

People have their preference in how code is styled and a good codebase is one that looks like a single person has written it.

Have a clear and consistent code style that is enforced via linters and formatters that work across the CLI and IDEs that the team uses. Use a configuration that is checked into version control to ensure consistency.

Duck typing enthusiasts can look away but please prefer strong typing (Typescript over Javascript, mypy on Python etc.). Your IDE suggestions and ease of exploration of language APIs will thank you, especially if your team aren’t experts at the language.

Build a codebase that has clean code architecture (layered, hexagonal, etc.). The codebase should clearly showcase design preferences (composition over inheritance) and even codify them through tests or fitness functions when possible.

When the code isn’t obvious, do not add comments. Write better tests and refactor your code.

Tooling and automation

Run formatters, linters, and tests automatically (using tools like pre-commit and husky). Build CI/CD pipelines that are fast and reliable which provide meaningful feedback when things fail. Automatic deployments to non-prod environments. Automated rollbacks strategies when deploying to production. It’s 2025 and there are very few reasons to need downtime even when running most standard migrations. Build observability into your pipelines to help diagnose issues (like pipelines slowing down) quicker.

A local developer experience that is consistent with production (docker, docker compose, and vagrant environments for more bespoke OS’). Use scripts for common workflows (just, npm scripts).

Builds need to be deterministic and reproducible. Lock your dependencies and avoid dependency hell. This might not be a big deal to the experience of developers on a daily basis but add periodic checks for outdated or vulnerable dependencies (using tools like snyk).

Testing and verification

Automate your tests with good quality unit tests for your logic, integration tests for the boundaries and (hopefully consumer driven) contract tests for external APIs that together, make up a good test pyramid or test trophy. Do not chase test coverage numbers. Use coverage to catch critical paths that are not well tested. Flaky tests suck, please eliminate them like a plague. Make them easy to run using an obvious command (like just test, npm test or ./gradlew test).

Use test doubles when necessary. Mocks and stubs are required but try to be stateful when possible (the last bit is a debatable opinion; one of the few endless debates in this blog). Use snapshot tests when possible but do not abuse this technique.

Lint your code. Do so early. Add security linters (like bandit or semgrep)

Collaboration and documentation

Every project should have a README.md, a TL;DR of your quick start guide for developers. Add a CONTRIBUTION.md for guidelines on how people can be good contributors (do you practice trunk based development or git flow? The answer will not be obvious to everyone when starting off on the project). Set up PR templates and code review guidelines to help aid internal conversations.

Automate your setup. New developers on your team should be productive in less than 5 minutes of the repository checkout (including the time taken to download dependencies).

If you’re creating an SDK or API, please generate auto-generated documentation. Capture decisions as Architectural Decision Records (ADRs) and C4 diagrams. This makes continued context maintenance and acquiring historical context easier.

Team workflows and culture

Decide on Trunk Based Development (TBD) or Git Flow (GF). If you’re going with TBD, merge early and merge often but do so with feature toggles. If you’re going with GF, create short-lived feature branches.

Set up a culture of blameless retros to learn from your mistakes effectively.

Track tech debt actively on the backlog and manage it regularly. Acknowledge and prioritise debt alongside features.

Ensure the team is used to sharing feedback openly. Set up retrospectives as a group and time to introspect as individuals.

This might all sound obvious in hindsight. So, why doesn’t every team invest in it? In truth, many developers have never experienced what great DevEx feels like. They don’t know it can be better, or they’ve accepted the friction as normal. But once you’ve worked in an environment where someone has sweated the details - where every part of the workflow feels seamless - you can’t unsee it. You start to expect it. And that expectation changes everything.

What’s next?

Every team deserves a developer experience that brings out their best work. Start by imagining what “great” looks like for your codebase - your north star. Then chart a course. Build a roadmap. Rally others. The path from chaos to clarity is paved with small, deliberate steps.

Take 10 minutes today to write down your team’s DevEx wishlist. Start a conversation with the team: What’s slowing us down? Pick one thing from the DevEx stack and implement it this week.

Change starts with you.

In the next blog, we’ll dive into how AI coding assistants can help amplify your impact - accelerating code quality, catching issues early, and automating the boring stuff - so you can focus on what really matters: building things that matter.

Credits

Thanks to Vinayak Kadam for providing feedback and Priyadarshan Patil for requesting me to write about this, after my passionate filled monologue in a conversation about Developer Experience.

Transparency has been a cornerstone of Sahaj throughout my journey here. It is not just a value we champion but a principle deeply embedded in how we operate. But transparency is not as simple as it sounds — it comes with its own challenges and costs.

Examples of transparency include sharing all business related data openly internally such as salaries, revenue, forecasted incoming work and people etc.

Transparency promotes empowerment

Trust is essential at Sahaj. We believe in empowering everyone to help our business grow by making informed decisions. For this to work, everyone must have access to the vital information that shows them the bigger picture. We strive hard to avoid informational hierarchy, where some people have information and others don’t. We believe everyone has the right to access key information, which will make people see the bigger picture and make the right decisions to help grow the business.

Having access to this information along with the power to make important decisions to grow our business is how each of us walks the path toward becoming better CEOs and business leaders. Consequently, some of us (myself included) experience an increased sense of fulfilment and engagement. It enables each one of us to articulate our ideas, grow and build expertise in the areas we would like to while simultaneously helping our business grow. However, this empowerment also comes with responsibilities and challenges, as transparency requires effort and understanding to truly benefit the collective.

Balancing collective and individual needs

Transparency, like any valuable principle, comes at a cost. Informational transparency makes information available to everyone. While the context is provided internally, everyone will not have spent the same time to absorb and process the information on their end since their priorities are different. Broadly, folks primarily responsible for operations (business oriented roles), spend more time thinking about these things than folks primarily responsible for software delivery.

Transparency, like radical candor, requires looking beyond initial reactions to understand the deeper context. Let us understand this with an example.

An example through personal experience

Let’s talk about open salaries. The concept is simple, ensure that everyone inside the organisation knows how much everyone else makes. We do this to make sure everyone in the organisation gets paid appropriately. For me, personally, it has made it possible to spend less time worrying about whether I get paid fairly and more time in actually doing my best work.

On day one, transparency might feel overwhelming — like staring at a spreadsheet of numbers without context. As one spends more time in the organisation, understands contexts and the value an individual brings in, they are then able to better co-relate the number with value. At this point, a lot of things start making sense like why I get paid what I do and why others around me get paid a similar amount. There will also be a ton of things that will not make sense. This is a pivotal point because either I could make assumptions or ask for clarification. Assumptions often lead to frustration, a path I prefer to avoid. Much like many other Sahajeevis, I took the route of asking for more information. Sahaj is a pull-based organisation meaning we expect people to ask questions whenever they have them and those with more context will provide it. Over time, as you engage and ask questions, patterns emerge, and you start to see the bigger picture.

At some point, you will interview a candidate you really like. However, based on your understanding of how salaries work internally, you might think this is an offer we should not make (usually because their expected salaries are too high). You know there are internal mechanisms to handle such situations. Despite having “similar knowledge”, some of us will be comfortable moving forward and some of us will not. There are a couple of reasons why people react differently to the same information.

1. They are comparing the candidate to themselves.
2. They are unable to see the bigger picture, either because they haven’t fully understood the information or because they cannot see past self interest (#1)

True cost of transparency

When people understand the larger picture and prioritise the collective over immediate self-interest, transparency transforms from a burden into a powerful tool for running an effective business. This balance between discomfort and long-term growth is the true cost — and reward — of transparency.

While many people believe they want transparency, what they often seek is convenience. Convenience to have access to data and to be able to use it to make arguments that serve their self interest. This is a normal part of the journey we all go through in transparent organisations.

The true reason for needing transparency in an organisation is to help teach all of us how to effectively run a business. This requires us to be uncomfortable at times. Uncomfortable because we need to put a collective (our business) before ourselves. Uncomfortable because we have to admit the fact that at times, we want to think of ourselves first and that, as leaders, there are times we cannot or should not. Uncomfortable with the realisation that while we might want to be leading our business, there are moments when we aren’t ready to do so. What we all want at times is just comfort and to not have to think of the big picture.

When we join an organisation like Sahaj, we get access to information. At some point, the information will not make sense to us (based on our context, available information and/or knowledge). The only way to grow is to start a conversation with others and evolve our perspectives, which drives us to realisation. This state of confusion isn’t permanent since we will eventually realise new unanswered questions which require growth in our perspective.

Embracing the Challenge and Growth of Transparency

Transparency grants us access to important information, which challenges us to think critically and grow both personally and professionally. However, this growth often comes with moments of discomfort — times when we must confront opposing viewpoints or accept hard truths. And that’s okay. It’s okay to feel uncomfortable, to need time to process, and to engage in open dialogue to navigate these complexities.

The rewards of transparency are profound: freedom, greater knowledge, continuous learning, and meaningful growth. Yet, the cost of transparency is equally real. It requires us to embrace discomfort, confront differences, and invest time in understanding and resolving them.

If you find yourself wanting transparency but hesitating to embrace the effort it demands, perhaps what you’re truly seeking is convenience — the comfort of data that reinforces your own perspective. There’s no shame in admitting this; I, too, have found myself drawn to the easier path at times. It’s human nature. But real progress, like radical candor, requires a willingness to embrace discomfort and challenge ourselves to change.

Transparency is not just a value — it’s a practice. It asks us to think beyond ourselves, to see the bigger picture, and to lead with empathy and understanding. If we’re willing to pay its price, transparency can transform not just our organisations but also ourselves as leaders and contributors to a collective vision.

Thanks to Kshitij, Swapnil, Puneet, Geeta, and Priyank for their reviews and early feedback.

What is event driven?

This technique is a popular technique to avoid coupling in systems. These systems tend to eventually become good sources of data that the business would like to build data platforms, insights and models on.

This page exists to

1. Help understand the different patterns at a high level
2. Understand the implications on building data systems

Events vs Commands

An event is when a system wants to announce what has happened but not what is to be done. For example, a new insurance quote being generated is an event. It announces to the world that a quote has been generated but not what should happen as a result.

A command is when a system wants something to be done and is asking a system to do it. For example, an upstream system might ask the communications system to send an email with specific details and this is a command to the communications system.

Both of these are usually implemented as events on a queue. The primary differences are how they are named and what the intent is.

Different types of event driven patterns

Let’s start with an example to help visualise the problem in which the customer changes an address for their house insurance in an insurance provider’s system which leads to a new quote being generated. This quote needs to be sent back to the user via an email.

If the services are built as visualised with calls across services being made, the services will be tightly coupled in their flow (since customer management needs to know of the existence of the quoting system which in-turn needs to know about the existence and need for communication). Here is how that problem can be solved with event driven architectures.

Event notification pattern

In this pattern, a source system will send a “notification” to all other systems that something has happened. The consumer needs to setup an event listener and figure out how to react to it. An example of this can be seen by the customer management system generating

Since the events do not have any information about what has changed, the downstream systems still need to call the upstream system to understand the details of what has changed to take action on the changes.

Here are a couple of versions of the customer changed event. The first version is one where the customer address changed event could include only the ID of the customer who’s address has changed. For every other part of the information (including what’s changed), the downstream systems need to contact the customer mamangement service.

Of course, these additional questions could be included in the event notification because they are related to the core event itself. There will always be some fields that a downstream system might need that are not directly part of the event but are required by the downstream system.

Advantages of using Event Notification

Systems built are decoupled. When there’s other actions that need to be made based on an address being changed, it’s easy to add another system to take action on this event without changes being required on the customer management side.

Downsides of using Event Notification

Systems built will be devoid of any behavior and there is no easy way to trace what happens downstream. There is no easy way to trace all the changes that happen in the code (by looking at the source code) to understand the list of changes that happen when the user changes their address.

Distributed tracing systems like zipkin aim to address these challenges by allowing visualisation of flows on environments with a full setup. Code can be traced by using mono-repos with the event names being the same across services. These are techniques to deal with the inability to trace code/flows across systems and while neither of them are as effective as tracing usages of your code, they help drive a balance between decoupling and ease of use.

Even when all the related information to the event have been added into the event payload, there will still be a need for downstream systems to require information. This means additional API calls will have to be made to the upstream system. As more downstream systems subscribe to a particular event, the upstream system will be under higher load to provide access to information and the downstream system’s availability is depedent on the upstream system.

Event carried state transfer pattern

Event carried state transfer (or ECST, for short) is sends all information related to the domain object in the event to avoid Event Notification’s need for call backs for additional information.

Downstream systems need to store the parts of the information they need for their usecase. If a difference between the old and new data is required, the data-structures chosen should make calculating differences easier.

Advantages of using ECST

Systems using this pattern have a lower dependence on their upstream services and thus have higher availability.

Downsides of using ECST

The higher availability comes at the cost of making the system eventually consistent. The data will also have higher replication.

Event sourcing

An event sourced system is one where the events are stored on an event store/event log and where the current application state can be completely recreated based on the event store.

The event store is an append only log of events that have occurred and, in the example, the customer DB is an example of a snapshot. A snapshot is required for enhancing the performance of your store (since it stores the current state of your system for quick access).

Both source control systems (like git, svn etc.) and financial accounting ledgers are good examples of event sourcing.

Advantages of using Event Sourcing

This system makes audit, debuggability and replayability simple. Such systems are great to recreate issues and understand what order things happened in. The ability to timetravel with data on a production system is quite useful. Concepts like branching is possible with data and what-ifs are easy to simulate to figure out the difference. Differences can then be applied through the creation of compensating actions.

Downsides of using Event Sourcing

This system will make event versioning mandatory. Interacting with external systems becomes more complicated since these calls are side-effects and event sourced systems should not cause this side-effect again when events are being replayed.

Command Query Responsibility Segregation pattern

Command Query Responsibility Segregation (or CQRS, for short) is a model in which reads and writes are separated. This allows scaling and optimisation reads and writes separately as per requirements.

Advantages of using CQRS

This pattern is rarely necessary but extremely useful to allow write heavy vs read heavy transactions to be scaled separately. The read side(s) can be optimised for the usecases they are being used for.

Downsides of using CQRS

Adds significant complexity building and maintaining a system.

Reading material

1. What is event driven?
2. The many meanings of Event-Driven Architecture?
3. Axon framwork - Framework for event driven, event sourced, CQRS powered applications in Java
4. Event Sourcing: A Practical Guide to Actually Getting It Done

What is DevOps and how is it so much bigger than automating deployments?

You know that a term you coined has made it mainstream when people use it regularly in conversations and rarely understand what you meant.

 — Martin Fowler (paraphrased from an in-person conversation)

Rouan summarises DevOps culture well in his post on Martin’s bliki. It is easy for developers to get disinterested with operational concerns. “It works on my machine” used to be a common phrase between developers in yesteryears. Some operations folks can also be less concerned by development challenges. Increased collaboration can help build a bridge in the gap between Developers and Operations team members and thus make your product better.

This increased collaboration has made observed requirements like system and resource utilisation monitoring, (centralised) logging, automated and repeatable deployments, no slow-flake servers etc. key parts of our products. Each of these improve the quality of life of your product either by directly benefiting the end user or making the system more maintainable for Developers and Operations users thus reducing the time to fix issues for end user issues. Developers and Operations folks are also first class users of your system. Their happiness (ease of debugging issues, deploying etc.) is a key part of your product’s success. It allows them to spend more time improving your product for paying end users.

What is MLOps?

MLOps is a culture that increases collaboration between folks building ML models (developers, data scientists etc.) and people who monitor these models and ensure everything is working as intended (operations). The observed requirements in your system will have some overlaps with what we have already talked about like system and resource monitoring, (centralised) logging, automated and repeatable deployments, automated creation of repeatable (non-snowflake) infrastructure etc. It will also include a few Data Platform specific observed requirements such as model and data versioning, data lineage, monitoring effectiveness of your model over an extended period of time, monitoring data drift etc.

Some tools/techniques to build a robust data platform

The need of every data platform is slightly different based on the challenges you are solving and the scale at which you operate. One of the platforms I’ve been working on produces 2TB of data every week. It didn’t take too much time for data storage costs to be the number 1 line item on our bill and we invested some time in optimising our storage and retention strategy. Other teammates have lowered data volumes and focus on reducing the cycle time for model creation. Your mileage may vary.

Based on our experience building data platforms over the past few years, here are a few tools we have used and things we have watched out for.

Data Storage

Choose a storage mechanism that provides cheap and reliable access to your data while meeting all legal requirements for your dataset. If you are in a heavily regulated environment (finance, medicine etc.), you might not be able to use the cloud for customer data. The techniques still remain similar. Partition your data based on access requirements and retention times. Archive data when you do not need it. Use features like push down predicates to efficiently read your data.

We recently wrote about data storage, versioning and partitioning which goes into great depth into this topic.

Job Scheduler/Workflow orchestrator

Your data pipelines will get complex over a period of time. Much like infrastructure as code, we would like our data pipelines in code. Apache Airflow is one of the tools that allows us to do this fairly easily. Sayan Biswas wrote about our airflow usage in 2019. Over the last few years, we have made dozens of improvements to the way we use Airflow. In a subsequent post in this series, we will talk through these improvements.

Monitoring and managing data processing costs

We spawn EMR clusters on demand and terminate them when jobs complete. A cluster runs only 1 spark job (and a few extra tasks for cleanups and reporting). If a job fails due to resource constraints, this helps isolate if another hungry job consumed too many resources before a scaling policy kicked in.

Each EMR cluster has an orchestrator node (AWS and Hadoop call them “master nodes”) and a group of core nodes (Hadoop calls them “worker nodes”). We request for on demand nodes for orchestrators and reserve the instances to reduce cost. We bid for spot instances for cores using a dynamic pricing strategy that is dependent on the current price. We have considered building a system that automatically switches instance types based on availability, price and stability in AWS but failures in spot bids are currently rare enough that it does not justify the cost of developing this feature.

We also monitor the resource utilisation of our spark jobs using Ganglia on AWS EMR. This tells us our CPU, memory, disk and network utilisation for our clusters. Since the information on Ganglia is lost when clusters are terminated, we run an EMR step to export a snapshot of Ganglia before the cluster terminates. This in conjunction with persisted spark history server data on AWS allows us to tune underperforming spark jobs. In a subsequent post, we will go into details of how to monitor your jobs effectively and tune them.

Monitoring the status of data pipeline jobs

Airflow creates EMR clusters and monitors each of the jobs. If a job fails, Airflow notifies us on a specific slack channel with links to the Airflow logs and AWS cluster.

Complex spark applications produce hundreds of megabytes of logs. These logs are distributed across the cluster and will be lost when the cluster is shut down. AWS EMR has an option to automatically copy the logs to S3 with a 2 minute delay.

We have tried using CloudWatch to index and analyse our spark logs but it was far too expensive. We also tried using a self hosted ELK stack but the cost of scaling it up for the volume of logs sent was too high. Dumping it on S3 and analysing it offline gave us the best cost to performance ratio.

To help reduce the time to fix an issue, when an issue is detected, the EMR cluster analyses its logs from YARN and publishes an extract onto slack as an attachment. Any further detailed analysis can be done on the logs in S3.

Monitoring data quality and data drift

Every time we write code, we run tests to ensure the code is safe to be deployed. Why don’t we do the same thing with data every time we access it?

When you first look at the data and build the model, you ensure the quality of the data used for training the model meets acceptable standards for your solution. Data Quality is measured by looking at the qualitative and quantitative pieces of your dataset. Over a period of time, these qualitative and quantitative attributes might drift causing adverse effects on your model. Thus, it is important to monitor your data quality and data drift. Data drift might be large enough that your model does not produce the right results any more or might be small enough to introduce a bias in your results. Monitoring these characteristics is key to producing accurate insights for your business.

Tools like Great Expectations and Deequ will ensure that your data is sound structurally and volumetrically. Deequ also has operators to look at rate of change of data which is a better expectation than having static thresholds on large volumes of data.

For example, given an employee salary database where the salary is nullable, a check to ensure no more than 100 employees out of the 1000 you currently have data for have no reported salary is bound to fail when the data volume increases significantly. If this check was to ensure no more than 10% of employees have no reported salary will work as the data scales as long as it scales evenly. Moving to a check that looks at rate of change of ratio of users not reporting a salary will be more robust. If the number changes significantly (up or down), it might mean that it’s time to tune your model since the source data is drifting away from when it was trained.

There are more complex examples on how we watch for data drift that will have to wait for a dedicated post.

The MLOps mindset

When our end users feel pain, we add new features to make their experience better. The same should be true for developers/operations experience (DevEx/OpsEx).

When it takes us longer to debug a problem or understand why a model did what it did, we improve our tooling and observability into our system. When it ran slower or was more expensive, we improved our observability to investigate inefficiencies quicker.

This has allowed us to grow our data platform 10x in terms of features and data volumes while reducing the time taken to produce insights for our end users by 98.75%, the cost to do so by 35% and not to mention a significant improvement in developer and customer experience.

Thanks to Jayant, Priyank, Anay and Trishna for reviewing drafts and providing early feedback. As always, Niki’s artwork wizardry is key!

In this post, we are going to use the terminology of AWS S3 buckets to store information. The same techniques can be applied on other cloud, non cloud providers and bare metal servers. Most setups will include a high bandwidth low latency network attached storage with proximity to the processing cluster or disks on HDFS if the entire platform uses HDFS. Your mileage may vary based on your team’s setup and use case. We are also going to talk about techniques which have allowed us to efficiently process this information using Apache Spark as our processing engine. Similar techniques are available for other data processing engines.

Managing storage on disk

When you have large volumes of data, we have found it useful to separate data that comes in from the upstream providers (if any) from any insights we process and produce. This allows us to segregate access (different parts have different PII classifications) and apply different retention policies.

We would separate each of these datasets so it’s clear where each came from. When setting up the location to store your data, refer to local laws (like GDPR) for details on data residency requirements.

Provider buckets

Providers tend to make their own directories to send us data. This allows them to have access over how long they want to retain data or if they need to modify information. Data is rarely modified but when it is, a heads up is given to re-process information.

If this was an event driven system, we would have different event types suggesting that the data from an earlier date was modified. Since the volume of data is large and the batch nature of data transfer on our platform, verbal/written communication is preferred by our data providers which allows us to re-trigger our data pipelines for the affected days.

Landing bucket

Most data platforms either procure data or produce it internally. The usual mechanism is for a provider to write data into its own bucket and give its consumers (our platform) access. We copy the data into a landing bucket. This data is a full replica of what the provider gives us without any processing. Keeping data we received from the provider separate from data we process and insights we derive allows us to

1. Ensure that we don’t accidentally share raw data with others (we are contractually obligated not to share source data)
2. Apply different access policies to raw data when it contains any PII
3. Preserve an untouched copy of the source if we ever have to re-process the data (providers delete data from their bucket within a month or so)

Core bucket

The data in the landing bucket might be in a format sub optimal for processing (like CSV). The data might also be dirty. We take this opportunity to clean up the data and change the format to something more suitable for processing. For our use case, a downstream pipeline usually consumes a part of what the upstream pipeline produces. Since only a subset of the data is read downstream by a single job, using a file format that allows optimized columnar reads helped us boost performance and thus we use formats like ORC and parquet in our system. The output after this cleanup and transformation is written to the core bucket (since this data is clean input that’s optimised for further processing and thus core to the functioning of the platform).

While landing has an exact replica of what the data provider gave us, core’s raw data just transforms it to a more appropriate format (parquet/ORC for our use case) and processing applies some data cleanup strategies, adds meta-data and a few processed columns.

Derived bucket

Your data platform probably has multiple models running on top of the core data that produce multiple insights. We write the output for each of these into its own directory.

Advantages of data segregation

1. Separating the data makes it easier to find the data. When you have terabytes or petabytes of information across your organization with multiple teams working on this data platform, it becomes easy to lose track of the information that is already available and it can be hard to find it if they are stored in different places. Having some way to find information is helpful. For us, separating the data by whether we get it from an upstream system, we produce it or we send it out to a downstream system helps teams find information easily.
2. Different rules apply to different datasets. You might be obligated to delete data from raw information you have purchased under certain conditions (like when they have PII). Rules for retaining derived data are different if it does not contain any PII.
3. Most platforms allow archiving of data. Separating the dataset makes it easier to archive different datasets. (we’ll talk about other aspects of archiving during data partitioning)

Data partitioning

Partitioning is a technique that allows your processing engine (like Spark) to read data more efficiently thus making the program more efficient. The most optimal way to partition data is based on the way it is read, written and/or processed. Since most data is written once and read multiple times, optimising a dataset for reads makes sense.

We create a core bucket for each region we operate in (based on data residency laws of the area). For example, since the EU data cannot leave the EU, we create a derived-bucket in one of the regions in the EU. Under this bucket, we separate the data based on the country, the model that’s producing the data, a version of the data (based on its schema) and the date partition based on which the data was created.

Reading data from a path like derived-bucket/country=uk/model=alpha/version=1.0 will give you a data set with columns year, month and day. This is useful when you are looking for data across different dates. When filtering the data based on a certain month, frameworks like spark allow the use of push down predicates making reads more efficient.

Data versioning

We change the version of the data every time there is a breaking change. Our versioning strategy is similar to the one talked about in the book for Database Refactoring with a few changes for scale. The book talks about many types of refactoring and the column rename is a common and interesting use case.

Since the data volume is comparatively low in databases (megabytes to gigabytes), migrating everything to the latest schema is (comparatively) inexpensive. It is important to make sure the application is usable at all points and that there is no point at which the application is not usable.

Versioning on large data sets

When the data volume is high (think terabytes to petabytes), running migrations like this is a very expensive process in terms of the time and resources taken. Also, the application downtime during the migration is large or there’s 2 copies of the dataset created (which makes storage more expensive).

Non breaking schema changes

Let’s say you have a dataset that maps the real names to superhero names that you have written to model=superhero-identities/year=2021/month=05/day=01.

+--------------+-----------------+
|  real_name   | superhero_name  |
+--------------+-----------------+
| Tony Stark   | Iron Man        |
| Steve Rogers | Captain America |
+--------------+-----------------+

The next day, if you would like to add their home location, you can write the following data set to the directory day=02.

+------------------+----------------+--------------------------+
|    real_name     | superhero_name |      home_location       |
+------------------+----------------+--------------------------+
| Bruce Banner     | Hulk           | Dayton, Ohio             |
| Natasha Romanoff | Black Widow    | Stalingrad, Soviet Union |
+------------------+----------------+--------------------------+

Soon after, you realize that storing the real name is too risky. The data you have already published was public knowledge but moving forward, you would like to stop publishing real names. Thus on day=03, you remove the real_name column.

+----------------+---------------------------+
| superhero_name |       home_location       |
+----------------+---------------------------+
| Spider-Man     | Queens, New York          |
| Ant-Man        | San Francisco, California |
+----------------+---------------------------+

When you read derived-bucket/country=uk/model=superhero-identities/ using spark, the framework will read the first schema and use it to read the entire dataset. As a result, you do not see the new home_location column.

scala> spark.read.
  parquet("model=superhero-identities").
  show()
+----------------+---------------+----+-----+---+
|       real_name| superhero_name|year|month|day|
+----------------+---------------+----+-----+---+
|Natasha Romanoff|    Black Widow|2021|    5|  2|
|    Bruce Banner|           Hulk|2021|    5|  2|
|            null|        Ant-Man|2021|    5|  3|
|            null|     Spider-Man|2021|    5|  3|
|    Steve Rogers|Captain America|2021|    5|  1|
|      Tony Stark|       Iron Man|2021|    5|  1|
+----------------+---------------+----+-----+---+

Asking Spark to merge the schema for you shows all columns (with missing values shown as null)

scala> spark.read.option("mergeSchema", "true").
  parquet("model=superhero-identities").
  show()
+----------------+---------------+--------------------+----+-----+---+
|       real_name| superhero_name|       home_location|year|month|day|
+----------------+---------------+--------------------+----+-----+---+
|Natasha Romanoff|    Black Widow|Stalingrad, Sovie...|2021|    5|  2|
|    Bruce Banner|           Hulk|        Dayton, Ohio|2021|    5|  2|
|            null|        Ant-Man|San Francisco, Ca...|2021|    5|  3|
|            null|     Spider-Man|    Queens, New York|2021|    5|  3|
|    Steve Rogers|Captain America|                null|2021|    5|  1|
|      Tony Stark|       Iron Man|                null|2021|    5|  1|
+----------------+---------------+--------------------+----+-----+---+

As your model’s schema evolves, using features like merge schema allows you to read the available data across various partitions and then process it. While we have showcased spark’s abilities to merge schemas for parquet files, such capabilities are also available with other file formats.

Breaking changes or parallel runs

Sometimes, you evolve and improve your model. It is useful to do parallel runs and compare the result to verify that it is indeed better before the business switches to use the newer version.

In such cases we bump up the version of the solution. Let’s assume job alpha v1.0.36 writes to the directory derived-bucket/country=uk/model=alpha/version=1.0. When we have a newer version of the model (that either has a very different schema or has to be run in parallel), we bump the version of the job (and the location it writes to) to 2.0 making the job alpha v2.0.0 and it’s output directory derived-bucket/country=uk/model=alpha/version=2.0.

If this change was made and deployed on 1st of Feb and this job runs daily, the latest date partition under model=alpha/version=1.0 will be year=2020/month=01/day=31. From the 1st of Feb, all data will be written to the model=alpha/version=2.0 directory. If the data in version 2.0 is not sufficient for the business on 1st Feb, we either run backfill jobs to get more data under this partition or we run both version 1 and 2 until version 2’s data is ready to be used by the business.

The version on disk represents the version of the schema and can be matched up with the versioning of the artifact when using Semantic Versioning.

Advantages

1. Each version partition on disk has the same schema (making reads easier)
2. Downstream systems can choose when to migrate from one version to another
3. A new version can be tested out without affecting the existing data pipeline chain

Summary

Applications, system architecture and your data always evolve. Your decisions in how you store and access your data affect your system’s ability to evolve. Using techniques like versioning and partitioning helps your system continue to evolve with minimal overhead cost. Thus, we recommend integrating these techniques into your product at its inception so the team has a strong foundation to build upon.

Thanks to Sanjoy, Anay Sathish, Jayant and Priyank for their draft reviews and early feedback. Thanks to Niki for using her artwork wizardry skills.

Life before version control

Before we can do that, it’s important to understand build process before we began on this journey.

Our build model for this project was branch based. Each environment maps to a branch (main -> dev, uat -> uat and production -> production). All other (feature) branches only ran the plan stage against the dev environment.

As you can notice, the configurations, secrets and keys are all maintained on the build agent. This means, every developer wanting to run plan and test their changes needs to replicate the terraform_variables directory. Any mistakes in doing so masks actual issues that your pipeline might face leading to delayed feedback.

Next, let’s look at what our codebase looked like

terraform
├── module-1
│   ├── backend.tf
│   ├── data.tf
│   ├── resources.tf
│   ├── provider.tf
│   └── variables.tf
├── module-2
│   ├── backend.tf
│   ├── data.tf
│   ├── resources.tf
│   ├── provider.tf
│   └── variables.tf
└── scripts
    └── provision
        ├── apply.sh
        ├── init.sh
        └── plan.sh

The provisioning scripts help us consistently run different stages across modules. Each module is an independent area of our infrastructure (such as core networking, HTTP services etc.)

Each of the provisioning scripts accepted a WORKSPACE_NAME (branch for execution that maps to the environment terraform is running for) and MODULE_NAME (module being executed).

init.sh ran the terraform init stage of the pipeline downloading the necessary plugins and initializing the backend

#!/bin/bash
set -e

cd $MODULE_NAME

echo "init default.tfstate"
terraform init -backend-config="key=default.tfstate"

echo "select or create new workspace $WORKSPACE_NAME"
terraform workspace select $WORKSPACE_NAME || terraform workspace new $WORKSPACE_NAME

echo "init $MODULE_NAME/terraform.tfstate"
terraform init -backend-config="key=$MODULE_NAME/terraform.tfstate" -force-copy -reconfigure

plan.sh ran the terraform plan stage allowing users to review their changes before applying them.

#!/bin/bash
set -e

cd $MODULE_NAME

echo "select or create new workspace $WORKSPACE_NAME"
terraform workspace select $WORKSPACE_NAME || terraform workspace new $WORKSPACE_NAME

echo "plan with var file ~/terraform_variables/$WORKSPACE_NAME/$MODULE_NAME.tfvars"
terraform plan -var-file=~/terraform_variables/$WORKSPACE_NAME/$MODULE_NAME.tfvars -out=$MODULE_NAME.tfplan -input=false

apply.sh applied the changes onto an environment. Developers do not run this command from local to ensure consistency on the environment

#!/bin/bash
set -e

cd $MODULE_NAME

echo "select or create new workspace $WORKSPACE_NAME"
terraform workspace select $WORKSPACE_NAME || terraform workspace new $WORKSPACE_NAME

echo "apply with var file ~/terraform_variables/$WORKSPACE_NAME/$MODULE_NAME.tfvars"
terraform apply -var-file=~/terraform_variables/$WORKSPACE_NAME/$MODULE_NAME.tfvars -auto-approve

Version controlling configuration

We moved the variables into the config directory by making a directory for every branch for each of the 3 environments we had.

terraform
├── config
│   ├── main
│   │   ├── module-1.tfvars
│   │   └── module-2.tfvars
│   ├── production
│   │   ├── module-1.tfvars
│   │   └── module-2.tfvars
│   ├── uat
│   │   ├── module-1.tfvars
│   │   └── module-2.tfvars
├── module-1
│   └── ...
├── module-2
|   └── ...
└── scripts
    ├── provision
    │   ├── apply.sh
    │   ├── functions.sh
    │   ├── init.sh
    │   └── plan.sh
    └── test_variable_names.sh

According to terraform’s documentation, you can export a variable that your terraform codes need with a prefix of TF_VAR.

functions.sh provides convenience functions to read the configuration and secrets.

#!/bin/bash

function fetch_variables() {
    workspace_name=$1
    module_name=$2

    echo $(cat ../config/$workspace_name/$module_name.tfvars | sed '/^$/D' | sed 's/.*/TF_VAR_& /' | tr -d '\n')
}

fetch_variables read the tfvars file, removes empty lines (that were added for readability), prefixed the name with TF_VAR and joined all entries into a single line. The string this method returns can be used as a prefix to the terraform command while running plan and apply making them environment variables.

Updated plan and apply scripts are placed in the secrets management section for brevity

Testing configuration files

The only limitation is that none of these variables can have a hyphen in the name because of shell variable naming rules. As with any potential mistake, a test providing feedback helps protect you from run time failures. test_variable_names.sh does this check for us.

#!/bin/bash

function parse_and_test_properties_entries() {
    prop=$1
    if [[ "$prop" == "" || $prop = \#* ]]; then
        continue
    fi

    key="$(cut -d'=' -f1 <<<"$prop")"
    if [[ $key =~ "-" ]]; then
        echo "$filename contains \"$key\" which contains a hyphen"
        exit 1
    fi
}

function parse_file() {
    filename=$1
    OLD_IFS=$IFS
    props=$(cat $filename)

    IFS=$'\n'
    for prop in ${props[@]}; do
        parse_and_test_properties_entries $prop
    done
    IFS=$OLD_IFS
}

base_dir="config"
for sub_dir in $(find $base_dir -mindepth 1 -maxdepth 1 -type d); do
    workspace_name=${sub_dir#"$base_dir/"}

    for input_file in config/$workspace_name/*.tfvars; do
        parse_file $input_file
    done

    echo "All variables are named correctly in config/$workspace_name"
done

Version controlling secrets

Secrets like passwords can be version controlled in a similar way though they require encryption to keep them safe. We’re using OpenSSL with a symmetric key to encrypt our secrets. Each secret is put into a tfsecrets file (internally a property file just like tfvars files for configuration). When encrypted, the file will have an extension of .tfsecrets.enc. When the plan or apply stages are executed, files are decrypted in memory (and not on disk, for security reasons) and used the same way.

functions.sh gets a new addition to support reading all secrets

function fetch_secrets() {
    workspace_name=$1
    module_name=$2
    secret_key_for_workspace=$(eval "echo \$SECRET_KEY_$workspace_name")
    echo $(openssl enc -aes-256-cbc -d -in ../config/$workspace_name/$module_name.tfsecrets.enc -pass pass:$secret_key_for_workspace | sed '/^$/D' | sed 's/.*/TF_VAR_& /' | tr -d '\n')
}

The astute amongst you probably noticed that we’re using OpenSSL v1.0.2s because v1.1.x changes the syntax on encryption/decryption of files. Also, you might have noticed the use of environment variables like SECRET_KEY_main, SECRET_KEY_uat and SECRET_KEY_production as the encryption keys. These values are stored on our CI server (in our case GitLab) which makes these values available to our CI agent during execution.

For local development, we have scripts to encrypt and decrypt configuration files either one at a time or in bulk per environment. It’s worth noting that re-encryption of the same file will show up on your git diff since the encrypted file’s metadata changes. Only check in encrypted files when their contents have changed (helping you debug future issues)

encrypt.sh takes SECRET_KEY as an environment variable for making local usage easier.

#!/bin/bash
set -e

if [ -z "$SECRET_KEY" ]; then
    echo "Set a SECRET_KEY for \"$WORKSPACE_NAME\" encryption"
    exit 1
fi

function encrypt_file() {
    input_file=$1
    target_file="$input_file.enc"
    echo "Encrypting $input_file to $target_file"
    openssl enc -aes-256-cbc -salt -in $input_file -out $target_file -pass pass:$SECRET_KEY
    rm -f $input_file
}

if [ -z $1 ]; then
    echo "Usage:"
    echo "  ./scripts/encrypt.sh <filePathFromProjectRoot>"
    echo "  ./scripts/encrypt.sh all"
    exit 2
elif [ "$1" == "all" ]; then
    for input_file in config/$WORKSPACE_NAME/*.tfsecrets; do
        encrypt_file $input_file
    done
else
    encrypt_file $1
fi

decrypt.sh also takes the same SECRET_KEY as an environment variable for making local usage easier.

#!/bin/bash
set -e

if [ -z "$SECRET_KEY" ]; then
    echo "Set a SECRET_KEY for \"$WORKSPACE_NAME\" decryption"
    exit 1
fi

function decrypt_file() {
    input_file=$1
    target_file=${input_file%".enc"}
    echo "Decrypting $input_file to $target_file"
    openssl enc -aes-256-cbc -d -in $input_file -out $target_file -pass pass:$SECRET_KEY
    rm -f $input_file
}

if [ -z $1 ]; then
    echo "Usage:"
    echo "  ./scripts/decrypt.sh <filePathFromProjectRoot>"
    echo "  ./scripts/decrypt.sh all"
    exit 2
elif [ "$1" == "all" ]; then
    for input_file in config/$WORKSPACE_NAME/*.tfsecrets.enc
    do
        decrypt_file $input_file
    done
else
    decrypt_file $1
fi

Testing secret files

If all files for an environment aren’t checked with the same key, you’ll face a runtime error. Since files can be encrypted individually, you must test if all files have been encrypted correctly. This test is also useful when you’re rotating the SECRET_KEY for an environment.

test_encryption.sh needs SECRET_KEY_<env> values set so it can be executed locally.

#!/bin/bash

base_dir="config"

for sub_dir in $(find $base_dir -mindepth 1 -maxdepth 1 -type d); do
    workspace_name=${sub_dir#"$base_dir/"}
    password_var_name="\$SECRET_KEY_$workspace_name"
    secret_key_for_workspace=$(eval "echo $password_var_name")

    if [ -z "$secret_key_for_workspace" ]; then
        echo "Variable $password_var_name has not been set. Unable to test"
        exit 1
    fi

    for input_file in config/$workspace_name/*.tfsecrets.enc
    do
        openssl enc -aes-256-cbc -d -in $input_file -pass pass:$secret_key_for_workspace &> /dev/null
        if [ $? != 0 ]; then
            echo "Unable to decrypt $input_file with $password_var_name"
            exit 1
        fi
    done

    echo "Successfully decrypted all secrets in config/$workspace_name"
done

End result

Our final project structure contains the following files

terraform
├── config
│   ├── main
│   │   ├── module-1.tfvars
│   │   ├── module-1.tfsecrets.enc
│   │   ├── module-2.tfvars
│   │   └── module-2.tfsecrets.enc
│   ├── production
│   │   ├── module-1.tfvars
│   │   ├── module-1.tfsecrets.enc
│   │   ├── module-2.tfvars
│   │   └── module-2.tfsecrets.enc
│   ├── uat
│   │   ├── module-1.tfvars
│   │   ├── module-1.tfsecrets.enc
│   │   ├── module-2.tfvars
│   │   └── module-2.tfsecrets.enc
├── module-1
│   └── ...
├── module-2
|   └── ...
└── scripts
    ├── decrypt.sh
    ├── encrypt.sh
    ├── provision
    │   ├── apply.sh
    │   ├── functions.sh
    │   ├── init.sh
    │   └── plan.sh
    ├── test_encryption.sh
    └── test_variable_names.sh

plan.sh uses functions.sh to load configuration and secrets

#!/bin/bash
set -e

source $(dirname "$0")/functions.sh

cd $MODULE_NAME

echo "select or create new workspace $WORKSPACE_NAME"
terraform workspace select $WORKSPACE_NAME || terraform workspace new $WORKSPACE_NAME

echo "plan with var file config/$WORKSPACE_NAME/$MODULE_NAME.tfvars"
config=$(fetch_variables $WORKSPACE_NAME $MODULE_NAME)
secrets=$(fetch_secrets $WORKSPACE_NAME $MODULE_NAME)
eval "$secrets $config terraform plan -out=$MODULE_NAME.tfplan -input=false"

apply.sh uses functions.sh in a similar fashion

#!/bin/bash
set -e

source $(dirname "$0")/functions.sh

cd $MODULE_NAME

echo "select or create new workspace $WORKSPACE_NAME"
terraform workspace select $WORKSPACE_NAME || terraform workspace new $WORKSPACE_NAME

echo "apply with var file config/$WORKSPACE_NAME/$MODULE_NAME.tfvars"
config=$(fetch_variables $WORKSPACE_NAME $MODULE_NAME)
secrets=$(fetch_secrets $WORKSPACE_NAME $MODULE_NAME)
eval "$secrets $config terraform apply -auto-approve"

And thus, our terraform project requires no data from the CI agent and can be executed perfectly from any box as long as it has the latest code checked out and the correct version of terraform.

git config --global commit.gpgsign true

What if you have different signatures for your personal ID and your work ID?

First, you create multiple signatures. It is important that the email address in the signature is the same as the one for the user who has authored the commit. Run gpg -K --keyid-format SHORT to see all available keys. The output looks like

/Users/karun/.gnupg/pubring.kbx
-------------------------------
sec   rsa4096/11111111 2019-06-11 [SC]
      1234567890123456789012345678901211111111
uid         [ultimate] Karun Japhet <karun@personal.com>
ssb   rsa4096/22222222 2019-06-11 [E]

sec   rsa4096/33333333 2019-06-11 [SC]
      0987654321098765432109876543210933333333
uid         [ultimate] Karun Japhet <karunj@work.com>
ssb   rsa4096/44444444 2019-06-11 [E]

Fetch the ID for each of the signatures. The ID for the personal signature is 11111111 and that for the work signature is 33333333. To assign a signature to the repo, execute git config user.signingkey <ID>.

Personally, I have aliases for personal and work signatures and every time I checkout a project, run the alias once.

alias signpersonal= "git config user.signingkey 11111111 && git config user.email \"karun@personal.com\""
alias signwork    = "git config user.signingkey 33333333 && git config user.email \"karun@work.com\""

Run git log --show-signature to verify if a commit used the right signature. Happy commit-signing.

If your login just won’t work, try changing the following settings

Privacy Badger

Allow calls to accounts.google.com & apis.google.com

Firefox settings

Allow Third party trackers in Firefox through Settings > Privacy & Security > Cookies > Third-party trackers