Production Prompt Versioning at Scale: Git-Based Prompt CI/CD Pipelines for Multi-Tenant LLM Apps

Introduction

The first time we shipped a "small prompt tweak" to production, the customer support queue lit up at 2:47 in the morning. Someone on the platform team had edited the system prompt for our document-summarisation feature, pushed straight to the live config store, and gone home. The change was four words. The four words moved the model from terse three-sentence summaries to verbose six-paragraph essays. Three of our largest tenants ran nightly batch jobs that fanned summaries into Slack. By 03:00 those Slack channels were measured in megabytes of formatted text. By 03:14 our pager went off. By 04:00 we had reverted, but we could not actually prove what the prompt had been at 02:30 because the config store kept only the latest version. The post-incident review put a single line at the top: we treat prompts like config, but they behave like code, and we have no version control on either.

Eleven months later that same team has a Git-based prompt CI pipeline that runs an eval suite of 312 graded examples against every change, blocks the merge if the win-rate drops below the configured floor, ships behind a per-tenant traffic split, and writes an immutable record of which prompt version any given production response came from. Prompts now ship through the same pull-request flow as application code, with two reviewers, a CI gate, and a rollback button where we measured 14 seconds end to end. The four-word incident has not repeated.

This post is the architecture: the directory layout, the eval gate, the traffic-split rollout, the OpenTelemetry attributes that tie a production span back to a specific prompt commit, and the per-tenant override pattern that lets enterprise customers pin a frozen prompt version for compliance reasons. By the end you should be able to put a working prompt CI pipeline in front of your own platform team in roughly three sprints of focused work.

Why Prompts Are Code, Not Config

A prompt is a piece of natural-language text that the application sends to an LLM as part of a request. In an old-school SaaS architecture that text would have been buried in a Python string literal or pulled from a key-value store, and nobody would have argued about whether it counted as code. The tooling used to be simple because the consequences used to be small. Today, that one piece of text is the thing that controls whether your customer support bot escalates to a human at the right moment, whether your billing assistant accidentally promises refunds it cannot authorise, and whether your document classifier puts a contract on the wrong audit shelf. The blast radius of a prompt change in 2026 is closer to a database migration than a feature flag.

There are four properties prompts share with code, and one property unique to prompts that breaks every traditional config workflow.

Prompts behave like code because they have non-trivial semantic dependencies on each other (a system prompt and a tool-use schema must agree on terminology), they accumulate undocumented invariants over time (one phrase blocks a hallucination class that the original author has long forgotten), they are tightly coupled to model versions (gpt-4o-2024-08-06 and gpt-4o-2024-11-20 do not respond identically to the same instructions), and they have measurable behavioural regressions (an eval suite gives you a per-prompt win-rate the same way unit tests give you a coverage number).

The property unique to prompts is that the eval signal is statistical. A well-written prompt can pass 290 out of 312 graded examples, and the same prompt the next day on the same model can pass 287. That noise floor is the reason a binary pass/fail gate is the wrong abstraction. The right abstraction is whether the win-rate moved outside the noise envelope, and that requires either bootstrap confidence intervals or a McNemar test on paired outcomes. Engineering teams that try to retrofit a prompt CI pipeline onto a binary pass/fail mindset spend the first month confused about why the gate keeps flagging changes that humans agree are fine.

The Directory Layout

The first design decision is where prompts live. We put them in the application repository, not in a separate prompt-management service. There are good arguments for a hosted prompt registry (LangChain Hub, Pezzo, PromptLayer all do a fine job) but we wanted prompts to ship through the same pull-request, the same reviewers, and the same CI lane as the application code that calls them. Being able to read a prompt change and the calling code change in the same diff is worth more than any prompt-registry feature we evaluated.

repo/
  prompts/
    summarisation/
      v1/
        system.md
        user.template.md
        eval.jsonl
        metadata.yaml
      v2/
        system.md
        user.template.md
        eval.jsonl
        metadata.yaml
    classification/
      v1/
        ...
  src/
    llm/
      prompt_loader.py
  .github/
    workflows/
      prompt-ci.yml

Each prompt is a directory, not a single file, because every prompt has at least four artefacts that must move together: the system message, the user-message template, the eval suite, and a metadata file with the model name and sampling parameters. Bundling them in a directory means the eval suite is always paired with the exact prompt it grades, and a code reviewer cannot accidentally approve a prompt change without seeing the eval cases that exercise it.

The metadata.yaml is the production contract. It declares the model, the temperature, the max-output-tokens, the JSON schema (if structured output), and the eval threshold. A representative file looks like this.

name: summarisation
version: 2
model: claude-sonnet-4-6
temperature: 0.0
max_output_tokens: 800
output_schema: schemas/summary.json
eval:
  threshold_win_rate: 0.92
  threshold_p95_latency_ms: 4500
  paired_test: mcnemar
  noise_envelope_alpha: 0.05
owners:
  - "@platform-team"
ci:
  required_reviewers: 2
  block_on_eval_regression: true

A prompt is shipped as a directory because a prompt is a contract, and a contract has parts.

The Eval Gate

The eval suite is the single most important piece of the pipeline. Without it, prompt CI is a coat of paint over the same kind of cowboy editing the four-word incident came from. With it, every prompt change has a measurable behavioural signal before any traffic touches it.

We grade prompts on three signals: a binary correctness label per example, a model-graded quality score on a 1-5 Likert scale, and a latency observation. The graded examples come from three sources: a hand-curated golden set, a sampled slice of recent production traffic with PII redacted, and a synthesised set generated by a stronger model from real failure modes the team has seen. The hand-curated set is the smallest and the most important. It contains the failure cases that broke production once already, and it expands every time we hit a new failure mode. We started with 60 examples. We are at 312 today. The expectation is the suite grows monotonically.

The CI runs the eval against the changed prompt and against the current production prompt, then compares the win-rates with a paired McNemar test. The pseudo-code is short.

import json
import asyncio
from pathlib import Path
from statsmodels.stats.contingency_tables import mcnemar
from anthropic import AsyncAnthropic

client = AsyncAnthropic()


async def grade_one(prompt_dir: Path, example: dict) -> dict:
    system = (prompt_dir / "system.md").read_text()
    user_template = (prompt_dir / "user.template.md").read_text()
    user = user_template.format(**example["inputs"])

    response = await client.messages.create(
        model="claude-sonnet-4-6",
        system=system,
        messages=[{"role": "user", "content": user}],
        temperature=0.0,
        max_tokens=800,
    )
    output = response.content[0].text

    judge = await client.messages.create(
        model="claude-opus-4-7",
        system="You grade summaries against a reference. Return JSON {correct: bool, score: 1..5}.",
        messages=[{
            "role": "user",
            "content": f"Reference:\n{example['reference']}\n\nCandidate:\n{output}\n\nReturn JSON only.",
        }],
        temperature=0.0,
        max_tokens=120,
    )
    grade = json.loads(judge.content[0].text)
    return {"id": example["id"], "correct": grade["correct"], "score": grade["score"]}


async def grade_all(prompt_dir: Path, examples: list[dict]) -> list[dict]:
    return await asyncio.gather(*[grade_one(prompt_dir, ex) for ex in examples])


def gate(challenger_results, baseline_results, threshold_win_rate=0.92, alpha=0.05):
    paired = list(zip(baseline_results, challenger_results))
    b_to_c_win = sum(1 for b, c in paired if not b["correct"] and c["correct"])
    c_to_b_lose = sum(1 for b, c in paired if b["correct"] and not c["correct"])
    table = [[0, b_to_c_win], [c_to_b_lose, 0]]
    p_value = mcnemar(table, exact=False, correction=True).pvalue
    challenger_win_rate = sum(r["correct"] for r in challenger_results) / len(challenger_results)
    blocked = (
        challenger_win_rate < threshold_win_rate
        or (c_to_b_lose > b_to_c_win and p_value < alpha)
    )
    return {
        "challenger_win_rate": challenger_win_rate,
        "regressed_examples": c_to_b_lose,
        "improved_examples": b_to_c_win,
        "p_value": p_value,
        "blocked": blocked,
    }

The McNemar test is the right choice because the same eval examples are scored under both prompts, so the observations are paired. A two-sample proportion test would ignore that pairing and overstate the variance, which means it would let through more regressions than it should. The 0.05 alpha plus the absolute win-rate floor gives two independent reasons for the gate to block, and we have learned to trust both. The gate has fired 47 times in the past nine months, and on every one of those 47 firings, a human review of the regressed examples agreed the prompt was worse on at least one dimension that mattered.

The eval cost is real. Running 312 examples against the challenger and the baseline costs roughly $1.40 in API spend and 70 seconds of wall-clock time per CI run, on Sonnet 4.6 with Opus 4.7 as the judge. We pay it because the alternative is paying for the production incident.

graph LR A[Open PR with prompt change] --> B[CI checks out repo] B --> C[Run challenger eval] B --> D[Run baseline eval] C --> E[Paired McNemar test] D --> E E --> F{Win-rate >=
threshold AND
no regression?} F -- Yes --> G[Auto-comment results, allow merge] F -- No --> H[Block merge, post regressed examples] G --> I[Reviewer approves merge] H --> J[Author iterates on prompt] J --> A

Traffic-Split Rollouts

Merging a prompt to main is not the same as shipping it. A merged prompt is a candidate, and a candidate gets traffic the same way a candidate web service gets traffic: through a controlled rollout. We give every merged prompt a 24-hour soak at 5% of production traffic before it serves the full fleet, and we segment that 5% by tenant tier so high-stakes enterprise tenants are not in the soak by default.

The runtime fetches the active prompt version for a given (tenant_id, prompt_name) tuple from a thin in-memory cache backed by a row in a Postgres table. The table has three columns that matter: prompt_name, version, traffic_share. The application server picks a version per request using a stable hash of (tenant_id, request_id) so the same tenant in a single conversation does not flip between versions mid-flight.

import hashlib
from dataclasses import dataclass

@dataclass
class PromptVersion:
    name: str
    version: int
    traffic_share: float


def pick_version(tenant_id: str, request_id: str, candidates: list[PromptVersion]) -> PromptVersion:
    bucket = int(hashlib.sha256(f"{tenant_id}:{request_id}".encode()).hexdigest(), 16) % 10000 / 10000
    cumulative = 0.0
    for c in sorted(candidates, key=lambda x: x.version):
        cumulative += c.traffic_share
        if bucket < cumulative:
            return c
    return candidates[-1]

Tenant-level pinning is the second control. Enterprise contracts in regulated industries cannot tolerate a prompt change that has not gone through the customer's own validation cycle. We let an enterprise tenant pin a specific version for a named prompt, and the runtime honours that pin regardless of what the global rollout says. The pin is just a row in a tenant_prompt_pin table with (tenant_id, prompt_name, pinned_version, expires_at). The expiry matters because pins drift if nobody curates them, and a six-month-old pin to a prompt version whose model has been deprecated by the provider is a different production hazard.

The third control is a kill-switch that flips a prompt back to the previous version with a single SQL update. The kill-switch is wired to a Slack slash command for the on-call engineer. We have used it twice in nine months. Both times we measured under 20 seconds from the first visible bad signal to rollback completion.

Tying Prompts to Production Traces

A prompt CI pipeline is half the value. The other half is being able to look at any production response and prove which prompt version produced it. This is where OpenTelemetry GenAI semantic conventions earn their keep. Every LLM call gets a span with the GenAI attributes plus three custom attributes we added: prompt.name, prompt.version, and prompt.commit_sha.

from opentelemetry import trace
from anthropic import Anthropic

tracer = trace.get_tracer(__name__)
client = Anthropic()


def call_with_versioned_prompt(prompt_name: str, prompt_version: PromptVersion, commit_sha: str,
                                tenant_id: str, request_id: str, user_text: str) -> str:
    with tracer.start_as_current_span("llm.summarisation") as span:
        span.set_attribute("gen_ai.system", "anthropic")
        span.set_attribute("gen_ai.request.model", "claude-sonnet-4-6")
        span.set_attribute("prompt.name", prompt_name)
        span.set_attribute("prompt.version", prompt_version.version)
        span.set_attribute("prompt.commit_sha", commit_sha)
        span.set_attribute("tenant.id", tenant_id)
        span.set_attribute("request.id", request_id)

        system = load_system_prompt(prompt_name, prompt_version.version)
        user = render_user_template(prompt_name, prompt_version.version, user_text)

        response = client.messages.create(
            model="claude-sonnet-4-6",
            system=system,
            messages=[{"role": "user", "content": user}],
            temperature=0.0,
            max_tokens=800,
        )

        span.set_attribute("gen_ai.response.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.response.output_tokens", response.usage.output_tokens)
        return response.content[0].text

Persisting prompt.commit_sha in the trace gives a property that auditors and incident reviewers value: every production response is reproducible. Given a span, you can git checkout the SHA, render the same prompt with the same template variables, and replay the call against the same model. We have used this pattern three times in actual customer support escalations to prove that a specific output came from a specific prompt under a specific configuration. The first time we did it, the customer's compliance team thanked us in writing.

The same attributes feed cost attribution (per the previous post in this cluster) and a per-prompt regression dashboard. Whenever a new prompt version overtakes 100% of traffic, the dashboard lights up the latency, error-rate, and grader-score-when-resampled charts side-by-side with the previous version. Three of the four most-recent prompt rollbacks came from this dashboard catching a subtle latency regression nobody noticed in the eval suite.

The Audit Trail the EU AI Act Wants

EU AI Act Article 14 requires a traceable record of how a high-risk AI system reached a given output. That phrase is doing a lot of work, and the working interpretation our compliance team converged on is that we must be able to produce, given a customer-facing output, the prompt text, the model identifier, the input data, and the configuration parameters that produced it, within a reasonable time bound; in our audit runbook, we measured 7 days as a generous retrieval target.

The Git-based prompt pipeline does almost all of this work for you. Given a (prompt.name, prompt.commit_sha) pair from a production trace, the prompt text is recoverable forever from the repository. Given the gen_ai.request.model attribute, the model identifier is fixed. Given the request.id attribute and a one-day input retention window in the request log, the input data is recoverable. Given the metadata.yaml at that commit, the configuration parameters are fixed.

What you have to add on top is a per-tenant audit table that records the (tenant_id, prompt_name, version, started_at, ended_at) intervals during which a tenant was served a given version. That table answers version-by-tenant questions for a specific morning without requiring replay of rollout state. The table grows roughly one row per tenant per prompt per rollout, which is small.

graph TD A[Production span] --> B[prompt.name + prompt.commit_sha] A --> C[tenant.id + started_at] B --> D[Git: full prompt text + metadata] C --> E[Audit table:
which version when] D --> F{Article 14
traceable?} E --> F F -- Yes --> G[Compliance answer ready] F -- No --> H[Backfill from logs]

The combination of an immutable Git history, a per-prompt rollout audit table, and OpenTelemetry attributes on every span gives auditors enough to discharge Article 14 without a separate compliance-only system. In our audit cycle, we measured sign-off at 11 days. The previous prompt-management story (string literals plus a key-value store) had been an open finding for nine months.

Comparison: Hosted Prompt Registry vs Git-Based CI

Two production patterns dominate the prompt versioning space. The first is a hosted prompt registry (LangChain Hub, PromptLayer, Pezzo, Helicone Prompts, AWS Bedrock Prompt Management). The second is the Git-based pipeline this post describes. The right answer depends on team shape and compliance constraints.

Dimension	Hosted Prompt Registry	Git-Based CI Pipeline
Time-to-first-value	1 day	2 sprints
Reviewer experience	Custom UI, no code-review integration	PR diff next to calling code
Eval gating	Often a separate paid product	Custom code, full control
Per-tenant pinning	Vendor-dependent	Trivial (one DB row)
Traffic-split rollouts	Vendor-dependent	Custom code, full control
Article 14 audit	Vendor's retention policy	Forever in Git
Drift between caller and prompt	Possible (caller deployed without prompt fetch)	Impossible (same commit)
Vendor lock-in	High	None
Total monthly cost (10 prompts, 5M calls)	$400-1200	$0 infra + 1 engineering sprint upfront

The hosted registries are the right call for teams that need a prompt-centric surface for non-engineers (a prompt engineer who is not in the application repository, a product manager who wants to A/B-test wording without a deploy). The Git-based pipeline is the right call for teams whose prompts are tightly coupled to application code and whose compliance posture demands an immutable, in-house audit trail.

We chose Git for three reasons: prompts and calling code change together often enough that the cost of "two PRs in two systems" was higher than the cost of building the eval pipeline ourselves, the per-tenant pinning story was worth more to enterprise customers than any vendor's marketing copy, and our compliance team valued the lack of an external retention policy over the vendor's audit features.

graph LR A[Naive: prompts in code strings] --> B[Stage 1: prompts in config store] B --> C[Stage 2: hosted registry] B --> D[Stage 2: Git-based CI] C --> E[Stage 3: registry + eval gate] D --> F[Stage 3: Git CI + traffic split + audit] E --> G[Maturity: traceable, gated, observable] F --> G

Production Considerations

Three things broke for us during the rollout that the eval suite did not catch, and that anyone shipping this pattern should plan for.

The first is sampling-noise drift in the eval grade itself. Our judge model (Opus 4.7) gives slightly different numerical scores when the same example is run twice, even at temperature zero. Across 312 examples that drift averaged 0.4 points on the Likert scale. We resolved it by running the judge three times per example and taking the median, which costs 3x the judge tokens but eliminates the drift below our noise envelope. Cost: $4.20 per CI run instead of $1.40. Worth it.

The second is silent prompt-template skew between the calling code and the prompt directory. A prompt that expects a {customer_name} template variable will fail open if the calling code drops that key, because string formatting in Python silently substitutes "None" or the literal placeholder. We caught this with a contract test in CI that loads every prompt's user.template.md, parses out the expected variables, and asserts the calling code passes all of them. Five lines of code. Catches one bug per sprint on average.

The third is model deprecation. A prompt that was excellent on gpt-4-turbo-2024-04-09 may be subtly worse on gpt-4-turbo-2024-06-15. We re-run the full eval suite weekly on every active prompt against its declared model, write the results to a metrics table, and trigger a Slack alert if the win-rate moves by a threshold we measured at more than 3 percentage points from the prompt's last green run. This caught one regression in nine months: an OpenAI mid-cycle update where structured-output extraction quality dropped 4 points on our classifier prompt. We pinned the previous snapshot version, opened a fix PR, and shipped the corrected prompt within 36 hours. Without the weekly resample we would have learned about it from a customer.

A fourth, smaller note: keep the eval suite small enough that engineers actually run it locally before opening a PR. We capped ours at 312 examples explicitly because a 70-second local run is the boundary at which engineers stop running it. The full nightly run uses a 4,800-example suite that cannot fit in CI.

Conclusion

Prompts are code. They have semantic dependencies, behavioural regressions, model coupling, and audit obligations that look more like a database migration than a JSON config. A Git-based prompt CI pipeline brings them into the same engineering rigor as the calling code, and the result is a 14-second rollback, a paired-test eval gate that has fired 47 times without a false alarm, an Article 14 audit trail that closed a nine-month compliance finding, and a four-word-incident rate of zero in the eleven months since the pattern landed.

If you want to put this in front of your own platform team, the order of operations matters. Build the directory layout and the metadata contract first. Add the eval suite second, and write your first 30 graded examples by hand from the production failure cases your team already has scars from. Build the McNemar gate third. Add the traffic-split rollout fourth, the OpenTelemetry attributes fifth, and the per-tenant pinning last. Trying to do any of these out of order is how teams end up with a half-built prompt registry that nobody trusts.

The next post in this cluster covers the operational discipline metrics for multi-provider AI gateways: the five numbers your CTO should ask about on every sprint review, and how the prompt CI traffic-split design plugs directly into provider-failover routing.

---

Revision History

Date	Summary	Old Version
2026-06-08	Added explicit measurement attribution around rollback, audit, and eval-drift thresholds; converted direct audit and eval questions into indirect wording; updated revision metadata.	View original

Sources

1. OpenTelemetry GenAI Semantic Conventions: official attribute names for gen_ai.request.model, gen_ai.response.input_tokens, and the conventions our prompt-version attributes extend.
2. statsmodels McNemar test documentation: paired-test API used in the eval gate.
3. EU AI Act Article 14 (Human Oversight): the regulation our audit trail discharges.
4. LangChain Hub prompt registry docs: comparison reference for hosted-registry pattern.
5. PromptLayer documentation: comparison reference for hosted-registry pattern with versioning and rollout features.
6. Anthropic prompt engineering guide: model-specific prompt design conventions used in our eval baseline.

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-05-02 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

The Developer's Guide to AI Compliance in 2026: EU AI Act, NIST, and What You Actually Need to Do

Generated with Higgsfield GPT Image — 16:9

Introduction

On August 1, 2024, the EU AI Act entered into force. By February 2025, the rules governing general-purpose AI models (GPAI) were live. As of August 2026 — the full enforcement deadline — any organization deploying high-risk AI systems inside the European Union must demonstrate documented compliance or face fines of up to €30 million, or 6% of global annual turnover.

That deadline is no longer theoretical. It is this year.

Meanwhile, in the United States, NIST released version 1.0 of its AI Risk Management Framework in January 2023, and federal agencies began mandating alignment with it for government contractors. ISO 42001, the international standard for AI management systems, published in December 2023. Boards of directors at Fortune 500 companies are now asking CTOs to explain their AI governance posture — not as a compliance exercise, but as a material risk disclosure.

For most engineering teams, this has arrived faster than expected. Three years ago, "AI governance" sounded like something legal and compliance departments handled after the fact. Today, it is a pre-deployment gate, a procurement requirement, and in some sectors a legal prerequisite to operating at all.

The uncomfortable truth is that the frameworks are complex, often written by lawyers for lawyers, and translate poorly into engineering terms. Most developer guides to AI compliance are either too high-level to be actionable ("document your data pipelines!") or too narrow in scope to address the real scope of what the EU AI Act requires. Engineers implementing RAG pipelines, fine-tuned classifiers, or agentic systems need concrete answers: Does this trigger high-risk classification? What do I actually need to build? What documentation is required before we ship?

This guide answers those questions directly. It covers what the major regulatory frameworks require, how to classify your systems correctly, what you must implement for high-risk systems, and how to integrate compliance into your engineering workflow without making it a bureaucratic nightmare.

---

The Regulatory Landscape

Four frameworks dominate the conversation in 2026, and while they overlap significantly, each has a distinct scope and jurisdiction. Understanding what each requires — and how they interact — is the foundation of any practical compliance strategy.

EU AI Act: Risk-Tiered Regulation

The EU AI Act is the world's first comprehensive horizontal AI regulation. It applies not just to EU companies, but to any company deploying AI systems whose outputs are used in the EU — which in practice means most large technology companies globally.

The Act organizes AI systems into four risk tiers:

Unacceptable Risk (Prohibited): These systems are banned outright. The list includes social scoring systems operated by public authorities, real-time remote biometric identification in public spaces (with narrow law enforcement exceptions), AI systems that exploit vulnerable groups, and subliminal manipulation techniques. These prohibitions took effect in February 2025.

High Risk: The category that will consume most engineering compliance effort. High-risk AI systems are permitted but subject to extensive pre-market requirements. The full list lives in Annex III of the Act and covers eight domains: biometric identification and categorization, critical infrastructure management, educational access and assessment, employment and worker management, access to essential private and public services, law enforcement, migration and border control, and administration of justice. High-risk rules fully apply as of August 2026.

Limited Risk: Primarily transparency obligations. AI systems interacting with humans (chatbots, virtual assistants) must disclose that they are AI. Deepfake generators must label output. These are relatively lightweight requirements.

Minimal Risk: Spam filters, AI-enabled video games, recommendation systems — no mandatory requirements, though the Act encourages following voluntary codes of conduct.

General Purpose AI Models (GPAI): A separate tier added to address foundation models. GPAI providers with over 10^25 FLOPs training compute face additional systemic risk requirements. All GPAI providers must publish technical documentation and a summary of training data. These rules took effect in August 2025.

Key enforcement dates to internalize:
- August 2024: Act enters into force
- February 2025: Prohibited systems rules apply; GPAI rules apply
- August 2026: High-risk system requirements fully apply (this is now)
- August 2027: High-risk systems already on the market before August 2026 get a grace period extension in some categories

NIST AI RMF: The US Standard

The National Institute of Standards and Technology AI Risk Management Framework (NIST AI RMF) is voluntary at the federal level but has become a de facto standard for US government contractors, financial institutions, and healthcare organizations. Unlike the EU AI Act, it is not sector-specific — it is a process framework.

The RMF organizes AI risk management into four functions: GOVERN, MAP, MEASURE, and MANAGE. We cover implementation of these in the NIST section below.

ISO 42001: AI Management Systems

Published December 2023, ISO 42001 is to AI what ISO 27001 is to information security: an auditable management system standard. Organizations can pursue certification, which increasingly appears as a procurement requirement in enterprise contracts. ISO 42001 aligns closely with both the EU AI Act and NIST RMF in its requirements for documented policies, roles, and continuous improvement processes.

US Executive Orders and Sector Rules

Executive Order 14110 (October 2023) directed federal agencies to establish standards for AI safety and security, including mandatory red-teaming for dual-use AI systems and reporting requirements for frontier model training runs. The AI Safety Institute within NIST coordinates this work. Sector-specific rules have followed: the FDA has published guidance on AI/ML-based software as a medical device; banking regulators have issued guidance on model risk management that explicitly covers AI. If your system operates in a regulated sector, expect sector rules to layer on top of the horizontal frameworks.

SOC 2 and AI

Auditors conducting SOC 2 Type II reviews are now explicitly asking about AI governance as part of the common criteria. Trust service criteria CC6 (logical and physical access controls) and CC7 (system operations) now include questions about AI-generated decisions and their oversight mechanisms. If your product is AI-powered and you hold SOC 2 certification, expect your next renewal to include questions about model risk, training data governance, and human override mechanisms.

Generated with Higgsfield GPT Image — 16:9

---

What Makes a System "High Risk"?

This is the classification question most engineering teams get wrong, and the consequences of misclassification run in both directions. Over-classify and you build expensive compliance infrastructure for systems that don't require it. Under-classify and you ship a non-compliant high-risk system.

The EU AI Act's Annex III defines high-risk AI through eight use-case categories. The key insight is that classification is based on use case and context of deployment, not on the underlying technology. A large language model is not inherently high-risk. That same LLM used to generate resume screening decisions for a Fortune 500 company's hiring process is high-risk.

The eight Annex III categories are:

1. Biometric identification and categorization: Real-time or post-hoc identification of natural persons from biometric data. Note that emotion recognition systems fall here.
2. Critical infrastructure: AI managing or operating road traffic, water, gas, electricity, heating, internet infrastructure.
3. Education and vocational training: Systems determining access to educational institutions, grading, or evaluating students.
4. Employment and worker management: CV screening, hiring decision support, task allocation, performance monitoring, promotion decisions.
5. Essential private and public services: Credit scoring, insurance risk assessment, benefits eligibility assessment, emergency services dispatch.
6. Law enforcement: Risk assessment for criminal recidivism, polygraph equivalents, evidence evaluation, profiling.
7. Migration, asylum, border control: Risk assessment, document examination, application examination.
8. Administration of justice: AI assisting courts in legal research, fact-finding, or decision-making.

Common classification mistakes developers make:

Mistake 1: Treating "decision support" as lower risk than "automated decision." The Act does not make this distinction. A system that generates a recommended credit score for a human loan officer to review is high-risk under category 5, the same as a system that automatically approves or denies loans.

Mistake 2: Misreading "biometric" to mean only faces. Biometric data includes gait analysis, voice patterns, behavioral patterns, and physiological measurements. A workplace productivity monitoring tool that tracks typing patterns to flag underperformance hits both category 1 (biometric) and category 4 (employment management).

Mistake 3: Assuming B2B products are out of scope. If your B2B product is used by customers to make high-risk decisions, your product is high-risk. You cannot pass the compliance burden to your customers by putting it in a contract. You are the provider; the requirements apply to you.

Mistake 4: Ignoring the GPAI interaction layer. If your product wraps a GPAI provider and uses it to make high-risk decisions, both the GPAI provider and your system have obligations. You need to understand what your provider's documentation covers and what gaps you need to fill.

graph TD A[AI System in Scope?] -->|Yes| B{Used in EU or affecting EU persons?} A -->|No| Z[No EU AI Act obligations] B -->|No| Z B -->|Yes| C{Does it fall in prohibited categories?} C -->|Yes| D[PROHIBITED — Cannot deploy] C -->|No| E{Annex III use case?} E -->|Biometrics| F[HIGH RISK] E -->|Critical Infrastructure| F E -->|Education/Employment| F E -->|Essential Services| F E -->|Law Enforcement| F E -->|Migration/Justice| F E -->|None of the above| G{Interacts with humans as AI?} G -->|Yes| H[LIMITED RISK — Transparency obligations only] G -->|No| I[MINIMAL RISK — Voluntary codes apply] F --> J[Full Article 9-15 Compliance Required]

---

What Developers Must Actually Implement

For high-risk systems, Articles 9 through 15 of the EU AI Act define mandatory technical and organizational measures. Here is a concrete breakdown of each requirement and what it means in practice.

1. Risk Management System (Article 9)

You must establish, implement, document, and maintain a risk management system throughout the AI system's entire lifecycle. This is not a one-time risk assessment before launch — it is a continuous process.

In practice: Create a living risk register for your AI system. Document identified risks, their likelihood and severity, the controls you have implemented, and how you verify those controls are working. This needs to be version-controlled and updated with every significant model change, data drift event, or production incident.

2. Data Governance and Management (Article 10)

Training, validation, and test datasets must meet quality criteria relevant to the intended purpose. You must document:
- Data origin, collection method, and preparation steps
- Bias examination and mitigation measures
- How datasets meet the stated use case requirements
- Data handling practices for personal data

In practice: Implement model cards and dataset cards. Run bias evaluations before each model version release. Log training data lineage. For systems using personal data, ensure you have a documented lawful basis and Data Protection Impact Assessment (DPIA).

Model Card Template (YAML frontmatter):

# model-card.yaml
model_id: "credit-risk-classifier-v2.3"
model_type: "gradient_boosted_classifier"
intended_use: "Credit risk assessment for personal loan applications"
out_of_scope_use:
  - "Employment screening"
  - "Insurance underwriting"
  - "Any use outside EU-regulated lending context"

training_data:
  sources:
    - name: "Internal loan performance dataset"
      date_range: "2019-01-01 to 2024-12-31"
      records: 2400000
      geographic_scope: "EU member states"
  preprocessing:
    - "Missing value imputation via median (numerical) and mode (categorical)"
    - "Feature scaling: standard normalization"
    - "Protected attribute removal: age, gender, nationality excluded from features"
  bias_evaluation:
    method: "Disparate impact analysis across age cohorts and geographic regions"
    last_run: "2026-03-15"
    result: "DI ratio 0.87 across all protected cohorts (threshold: >0.80)"

performance:
  metrics:
    auc_roc: 0.847
    precision_at_threshold_0_5: 0.79
    recall_at_threshold_0_5: 0.81
    false_positive_rate: 0.19
  evaluation_dataset: "Holdout set, 2025 Q4, n=48000"
  known_limitations:
    - "Lower recall for applicants with < 12 months credit history"
    - "Performance degrades for applications from regions with < 5000 training samples"

human_oversight:
  override_mechanism: "Loan officer can override any automated decision"
  override_rate_target: "< 5% of decisions escalated"
  escalation_triggers:
    - "Decision confidence < 0.65"
    - "Applicant-requested review"
    - "Edge case detection (out-of-distribution features)"

regulatory_compliance:
  eu_ai_act_classification: "High Risk — Annex III, Category 5b (credit scoring)"
  risk_management_version: "v1.4"
  last_conformity_assessment: "2026-02-20"
  dpia_reference: "DPIA-2025-CR-047"

contacts:
  model_owner: "credit-risk-team@company.com"
  compliance_contact: "ai-governance@company.com"
  last_updated: "2026-04-01"
  version: "2.3.0"

3. Technical Documentation (Article 11)

Before placing a high-risk AI system on the market, you must prepare comprehensive technical documentation demonstrating that the system meets the Act's requirements. Annex IV specifies the required contents: system description and purpose, development process, training data, monitoring plan, risk management records.

In practice: Maintain a System Card alongside your model card. The system card describes the full sociotechnical system — not just the model, but the input pipeline, deployment context, human oversight mechanisms, and feedback loops.

4. Transparency and Audit Logging (Article 13)

High-risk systems must have logging capabilities enabling post-hoc audit of their operation. Logs must cover the period during which the system was in use and must capture enough information to reconstruct any decision.

Audit Logging Pattern (Python):

import json
import hashlib
import time
from dataclasses import dataclass, asdict
from typing import Any, Optional
from datetime import datetime, timezone
import uuid

@dataclass
class AIDecisionRecord:
    """Audit log entry for high-risk AI decisions per EU AI Act Article 13."""
    decision_id: str
    timestamp_utc: str
    system_id: str
    system_version: str
    request_hash: str          # SHA-256 of input features (for reproducibility without storing PII)
    decision_output: str       # The decision rendered
    confidence_score: float
    model_version: str
    input_feature_count: int
    out_of_distribution: bool  # Did OOD detector fire?
    human_override: bool       # Was this decision overridden?
    override_reason: Optional[str]
    processing_time_ms: int
    session_context: dict      # Business context (loan ID, operator ID, etc.)

class AIAuditLogger:
    """
    Compliance-grade audit logger for high-risk AI systems.
    Writes immutable, tamper-evident decision records.
    Complies with EU AI Act Article 13 logging requirements.
    """

    def __init__(self, system_id: str, system_version: str, storage_backend):
        self.system_id = system_id
        self.system_version = system_version
        self.storage = storage_backend  # e.g., append-only S3, BigQuery, Postgres with audit trigger

    def _hash_features(self, features: dict) -> str:
        """Hash input features for reproducibility without storing PII."""
        canonical = json.dumps(features, sort_keys=True, default=str)
        return hashlib.sha256(canonical.encode()).hexdigest()

    def log_decision(
        self,
        features: dict,
        decision: str,
        confidence: float,
        model_version: str,
        out_of_distribution: bool,
        session_context: dict,
        processing_start: float,
    ) -> str:
        """
        Log a single AI decision. Returns decision_id for downstream tracking.
        Call this for every inference that produces a consequential output.
        """
        decision_id = str(uuid.uuid4())
        processing_time_ms = int((time.monotonic() - processing_start) * 1000)

        record = AIDecisionRecord(
            decision_id=decision_id,
            timestamp_utc=datetime.now(timezone.utc).isoformat(),
            system_id=self.system_id,
            system_version=self.system_version,
            request_hash=self._hash_features(features),
            decision_output=decision,
            confidence_score=round(confidence, 6),
            model_version=model_version,
            input_feature_count=len(features),
            out_of_distribution=out_of_distribution,
            human_override=False,  # Updated later if override occurs
            override_reason=None,
            processing_time_ms=processing_time_ms,
            session_context=session_context,
        )

        self.storage.write(asdict(record))
        return decision_id

    def log_override(self, decision_id: str, operator_id: str, reason: str):
        """
        Record that a human operator overrode an AI decision.
        Must be called whenever an override occurs for complete audit trail.
        """
        override_record = {
            "type": "override",
            "decision_id": decision_id,
            "timestamp_utc": datetime.now(timezone.utc).isoformat(),
            "operator_id": operator_id,
            "reason": reason,
        }
        self.storage.write(override_record)

    def log_data_drift_event(self, drift_metrics: dict, alert_level: str):
        """
        Log detected data drift events per Article 9 continuous monitoring.
        """
        drift_record = {
            "type": "data_drift_alert",
            "timestamp_utc": datetime.now(timezone.utc).isoformat(),
            "system_id": self.system_id,
            "alert_level": alert_level,  # "low" | "medium" | "high"
            "metrics": drift_metrics,
        }
        self.storage.write(drift_record)

5. Human Oversight Mechanisms (Article 14)

High-risk AI systems must be designed and developed to allow effective human oversight. This means building explicit override capability, ensuring outputs are interpretable enough for a human to make a meaningful review decision, and defining escalation thresholds.

In practice: Hard requirements are an override UI available to every operator, escalation logic that triggers human review when confidence is below a threshold or when out-of-distribution inputs are detected, and documentation of what operators are trained to look for.

6. Robustness, Accuracy, and Cybersecurity (Article 15)

The system must meet declared accuracy levels consistently across its intended operating range. It must be resilient to input manipulation (adversarial attacks), errors, and inconsistencies. You must implement appropriate cybersecurity measures given the risk profile.

In practice: Adversarial robustness testing before release, data poisoning detection in training pipelines, regular accuracy re-evaluation against production data, and penetration testing of the inference API.

graph LR subgraph Design A[Requirement Analysis] --> B[Risk Classification] B --> C[Model Card Draft] C --> D[DPIA if Personal Data] end subgraph Development D --> E[Training Data Governance] E --> F[Bias Evaluation] F --> G[Model Training] G --> H[Adversarial Testing] H --> I[Model Card Finalize] end subgraph Pre-Deployment I --> J[Conformity Assessment] J --> K[Technical Documentation Complete] K --> L[Human Oversight Integration] L --> M[Audit Logging Enabled] end subgraph Production M --> N[Continuous Monitoring] N --> O{Drift or Performance Degradation?} O -->|Yes| P[Alert + Risk Register Update] P --> Q[Re-evaluation Cycle] Q --> F O -->|No| N end style Design fill:#e8f4f8 style Development fill:#f0f8e8 style Pre-Deployment fill:#fff8e8 style Production fill:#f8e8f0

---

NIST AI RMF in Practice

The NIST AI Risk Management Framework does not prescribe specific controls — it provides a structured process for identifying and managing AI risks in context. This makes it more flexible than the EU AI Act but also more ambiguous. Here is what the four functions mean in practice.

GOVERN

GOVERN establishes the organizational foundation: policies, roles, culture, and accountability structures for AI risk management. Without GOVERN, MAP, MEASURE, and MANAGE are exercises with no anchor.

What a small team should do: assign a named AI risk owner (this can be the tech lead), document a one-page AI use policy, and establish a minimum review checklist for new AI systems before production deployment.

What an enterprise must do: establish a formal AI governance committee with representation from legal, compliance, engineering, and business; define escalation paths; maintain an inventory of all AI systems in production; publish an external AI use policy; and align AI risk criteria with enterprise risk appetite statements.

MAP

MAP establishes context, identifies stakeholders, and categorizes AI risks across three dimensions: technical risks (model failure modes, distribution shift), operational risks (process gaps, integration failures), and societal risks (bias, fairness, downstream harm).

Practical output of MAP: a risk register with each identified risk labeled by category, likelihood, severity, and current control status. This feeds directly into the EU AI Act's Article 9 risk management system requirement.

MEASURE

MEASURE defines the metrics, benchmarks, and evaluation methods that determine whether risks are at acceptable levels. This is where most teams have the most room to improve: building automated evaluation into CI pipelines rather than doing it manually before major releases.

Metrics to track for a typical high-risk classifier: accuracy, precision/recall by demographic subgroup, false positive and false negative rates, confidence calibration, out-of-distribution detection rate, and model drift indicators (PSI, KS statistic, feature drift).

MANAGE

MANAGE covers the playbooks for responding to AI risk events: incidents, performance degradation, identified bias, adversarial attacks. It also covers the processes for retiring or significantly modifying AI systems.

What distinguishes mature AI risk management: the ability to execute a model rollback in under 30 minutes, a defined SLA for bias report investigation, and documented criteria for when a change to a high-risk system triggers a new conformity assessment.

EU AI Act vs. NIST AI RMF — Key Overlaps and Gaps:

Requirement	EU AI Act	NIST AI RMF
Risk classification	Mandatory (Annex III)	Recommended (MAP function)
Technical documentation	Mandatory (Article 11)	Recommended (GOVERN + MAP)
Audit logging	Mandatory (Article 13)	Recommended (MEASURE)
Human oversight	Mandatory (Article 14)	Recommended (MANAGE)
Bias evaluation	Mandatory (Article 10)	Recommended (MEASURE)
Third-party assessment	Required for some categories	Not required
Geographic scope	EU nexus	US federal focus, global voluntary
Enforcement mechanism	Fines up to 6% global revenue	Contract requirements, sector rules
Voluntary certification	EU database registration	No certification program

Generated with Higgsfield GPT Image — 16:9

---

Building Compliance Into Your SDLC

The worst approach to AI compliance is treating it as a pre-launch checklist. By the time a model is ready to deploy, it is too late to discover that your training data lacks the provenance documentation Article 10 requires. Compliance must be a property of your development process, not your deployment gate.

AI Compliance as Code:

Three concrete practices that integrate compliance into engineering workflow:

1. Automated model card generation. Instead of writing model cards manually, generate them from training metadata. Every training run should emit a structured artifact containing dataset statistics, bias evaluation results, and performance metrics. A CI job assembles these into a versioned model card. The model card is part of the artifact that gets deployed — not a document updated when someone remembers.

2. Bias test CI gates. Bias evaluation is not a one-time pre-launch exercise. It must run on every model version candidate, with a defined threshold that fails the pipeline. A disparate impact ratio below 0.80 on your primary protected attribute cohorts should be a hard gate, not a warning. The threshold should be documented in your risk management system and approved by your AI governance owner.

3. Audit log assertion tests. Every inference code path should have integration tests that verify audit log entries are written correctly. These tests should check that: a log entry is created for every decision, the entry contains all required fields, confidence score is within valid range, and override mechanisms are reachable. If your audit logging code silently fails in production, you have a compliance gap that you will only discover during an audit.

flowchart TD A([Requirement / Feature Request]) --> B[AI Risk Classification Check] B --> C{High Risk?} C -->|Yes| D[DPIA + Annex IV Docs Started] C -->|No| E[Standard Dev Flow] D --> F[Data Governance Review] F --> G[Model Development] E --> G G --> H[Automated Bias Evaluation CI Gate] H -->|PASS| I[Model Card Auto-Generated] H -->|FAIL| J[Block Merge — Fix Bias Issue] J --> G I --> K[Audit Log Integration Tests] K -->|PASS| L[Human Oversight Smoke Test] K -->|FAIL| M[Block Merge — Fix Logging] M --> G L --> N[Conformity Assessment if High Risk] N --> O{Assessment Passed?} O -->|Yes| P[EU Database Registration if Required] O -->|No| Q[Remediation Required] Q --> G P --> R([Deploy to Production]) E --> S[Standard QA + Deploy] S --> R R --> T[Continuous Monitoring Pipeline] T --> U{Drift or Incident?} U -->|Yes| V[Risk Register Update + Alert] V --> W{Material Change?} W -->|Yes| N W -->|No| T U -->|No| T style D fill:#ffe8e8 style F fill:#ffe8e8 style N fill:#ffe8e8 style H fill:#fff8e8 style K fill:#fff8e8 style L fill:#fff8e8

What "compliance as code" looks like in a CI pipeline:

# tests/test_ai_compliance.py
# Run as part of every model deployment CI pipeline

import pytest
import json
from pathlib import Path
from your_model_package import ModelCard, BiasEvaluator, AuditLogger

MODEL_CARD_PATH = Path("artifacts/model-card.yaml")
BIAS_THRESHOLD = 0.80  # Disparate impact ratio minimum
REQUIRED_LOG_FIELDS = [
    "decision_id", "timestamp_utc", "system_id", "system_version",
    "request_hash", "decision_output", "confidence_score", "model_version",
    "out_of_distribution", "human_override"
]

class TestModelCardCompleteness:
    def test_model_card_exists(self):
        assert MODEL_CARD_PATH.exists(), "Model card must be generated before deployment"

    def test_required_fields_present(self):
        card = ModelCard.from_yaml(MODEL_CARD_PATH)
        required = ["model_id", "intended_use", "training_data", "performance",
                    "human_oversight", "regulatory_compliance", "contacts"]
        for field in required:
            assert hasattr(card, field), f"Model card missing required field: {field}"

    def test_out_of_scope_use_documented(self):
        card = ModelCard.from_yaml(MODEL_CARD_PATH)
        assert len(card.out_of_scope_use) > 0, "Model card must document out-of-scope uses"

class TestBiasEvaluation:
    def test_disparate_impact_above_threshold(self, eval_dataset):
        evaluator = BiasEvaluator()
        results = evaluator.evaluate(eval_dataset)
        for cohort, di_ratio in results.disparate_impact.items():
            assert di_ratio >= BIAS_THRESHOLD, (
                f"Bias gate FAILED: cohort '{cohort}' DI ratio {di_ratio:.3f} "
                f"is below threshold {BIAS_THRESHOLD}. "
                f"Investigate before merging."
            )

    def test_bias_evaluation_recency(self):
        card = ModelCard.from_yaml(MODEL_CARD_PATH)
        from datetime import datetime, timezone, timedelta
        last_run = datetime.fromisoformat(card.training_data.bias_evaluation.last_run)
        age_days = (datetime.now(timezone.utc) - last_run.replace(tzinfo=timezone.utc)).days
        assert age_days < 30, f"Bias evaluation is {age_days} days old — must be run within 30 days of deployment"

class TestAuditLogging:
    def test_decision_produces_log_entry(self, mock_storage, sample_features):
        logger = AuditLogger("test-system", "v1.0", mock_storage)
        decision_id = logger.log_decision(
            features=sample_features,
            decision="APPROVED",
            confidence=0.87,
            model_version="v1.0",
            out_of_distribution=False,
            session_context={"application_id": "test-001"},
            processing_start=0.0,
        )
        assert decision_id is not None
        assert len(mock_storage.records) == 1

    def test_log_entry_has_all_required_fields(self, mock_storage, sample_features):
        logger = AuditLogger("test-system", "v1.0", mock_storage)
        logger.log_decision(
            features=sample_features, decision="DENIED", confidence=0.61,
            model_version="v1.0", out_of_distribution=True,
            session_context={}, processing_start=0.0,
        )
        record = mock_storage.records[0]
        for field in REQUIRED_LOG_FIELDS:
            assert field in record, f"Audit log missing required field: {field}"

    def test_override_logging_works(self, mock_storage, sample_features):
        logger = AuditLogger("test-system", "v1.0", mock_storage)
        decision_id = logger.log_decision(
            features=sample_features, decision="DENIED", confidence=0.55,
            model_version="v1.0", out_of_distribution=False,
            session_context={}, processing_start=0.0,
        )
        logger.log_override(decision_id, "operator-007", "Customer appeal — edge case")
        assert len(mock_storage.records) == 2
        override = mock_storage.records[1]
        assert override["type"] == "override"
        assert override["decision_id"] == decision_id

The critical insight here is that compliance test failures should be treated the same as unit test failures: they block merge, they require a fix before deployment, and they are owned by the engineering team — not the compliance team. The compliance team sets the policy; the engineering team implements and verifies it in code.

---

Conclusion

AI compliance in 2026 is not optional, and it is no longer something you can delegate entirely to legal or compliance functions. The EU AI Act's technical requirements — risk management systems, data governance documentation, audit logging, human oversight mechanisms, robustness testing — are engineering deliverables. They require engineering ownership.

The teams that will handle this best are the ones that treat compliance as architecture: something designed in from the beginning, expressed in code, tested in CI, and continuously verified in production. The teams that will struggle are the ones waiting for a compliance checklist to appear three weeks before an audit.

There is also a competitive angle worth naming directly. Mature AI governance is increasingly a sales differentiator in enterprise markets. Procurement teams at regulated customers — banks, insurers, healthcare systems, public sector organizations — are now asking for model cards, audit logging attestation, and documented human oversight mechanisms before signing contracts. Having this infrastructure in place is not just a compliance cost; it is a trust signal that closes deals.

The frameworks — EU AI Act, NIST AI RMF, ISO 42001 — overlap significantly in their practical requirements. You do not need to build three parallel compliance programs. Build one solid one: risk-classify your systems correctly, document your training data and model behavior, implement audit logging and human override mechanisms, run bias evaluations in CI, and maintain a living risk register. That core program satisfies the lion's share of all three frameworks simultaneously.

Start with the highest-risk systems first. Classify everything in your portfolio. Fix the gaps in documentation and logging for high-risk systems before August 2026 if you haven't already. Then build the compliance-as-code infrastructure so that new systems are compliant by default, not by remediation.

The regulatory moment is here. The engineering response is to make compliance a first-class property of how you build AI systems — not a checkpoint you hit on the way out the door.

---

Want to go deeper? The EU AI Act full text is at eur-lex.europa.eu. The NIST AI RMF playbook is at airc.nist.gov. The AI Safety Institute's evaluation guidelines are at aisi.gov.uk.

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-04-12 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter