AI Governance Is a Measurement Problem Before It Becomes a Model Problem

Enterprise AI discussions usually begin with model choice and end with adoption anxiety. That is backwards. The first real question is not whether a model can draft an answer, summarize a ticket, or generate a patch. It is whether the organization can measure net value without hiding costs in another team’s backlog. Most AI programs fail the moment they are treated as speed features instead of operating changes.

The evidence base points in the same direction. A large field study of customer support agents found that a generative AI assistant increased issues resolved per hour by about 14% on average, with a 34% improvement for novice and lower-skilled workers and little effect on experienced workers (NBER). A BCG field study reported 12% more tasks completed, 25% faster, with higher quality on tasks inside the model’s frontier (SSRN). A METR randomized trial on experienced open-source developers found the opposite pattern in a different setting: AI made them about 19% slower (METR). A 2025 MIT NANDA report on enterprise pilots claimed that most pilots were still producing zero return (MIT NANDA report). Those four signals do not contradict each other. They describe a simple rule: AI value is highly context dependent, and local acceleration can be offset by transferred costs elsewhere in the workflow.

That is the governance problem. CFOs care about whether the program pays back. CTOs care about whether it can be run safely, repeated reliably, and debugged without heroics. Both roles need the same thing: a controlled trial design that separates direct gains from side effects.

Why local speed is not the same as net value

A team can get faster at drafting emails, answering support tickets, or writing boilerplate code and still become slower at delivering the business outcome. The missing link is always the same: the work does not end where the model output ends. Faster drafting can create more review work. Easier ticket responses can increase escalation quality requirements. Faster code generation can shift effort into testing, architecture review, security review, or bug triage. The apparent productivity gain is real only if the rest of the system can absorb it.

This is why CFOs and CTOs should avoid one-number metrics such as “hours saved,” “tokens used,” or “percentage of tasks automated.” Those numbers are inputs or local outputs, not value. A useful AI program changes the whole cost stack: cycle time, rework, defect leakage, customer satisfaction, incident load, review time, and manager attention. If the new workflow is only faster at the first step, the organization may simply be borrowing time from later steps and booking it as savings too early.

Transferred costs are easiest to miss when they move across functions. Support gains may show up as product backlog growth. Sales drafting gains may turn into legal review load. Coding assistant gains may increase QA or security burden. Finance sees a smaller labor line and thinks the program worked; operations sees more exception handling and knows the bill moved somewhere else.

Design the trial around the whole workflow

A controlled trial should compare end-to-end work, not a narrow demo. The unit of analysis should be a complete business task that has a clear start, finish, owner, and downstream consequence. For support, that might be a resolved case. For finance, a reconciled exception. For engineering, a merged change that survives review and testing. For operations, a ticket or request that reaches closure without hidden clean-up.

The cleanest trial design is a pre-registered before-and-after or holdout experiment with one control group and one treatment group. Keep the task mix stable. Keep the service-level policy stable. Keep the escalation policy stable. Then instrument the workflow at each stage:

• input volume and task type
• time to first draft or first answer
• human review time
• rework count
• escalation rate
• exception or rollback rate
• customer or internal satisfaction
• cost per completed task

That list matters because it reveals whether AI shifts effort or actually removes it. If drafting time drops while review time doubles, the program may still be negative. If ticket volume rises because the system makes it easier to answer low-value issues, the team may feel busier while net value stays flat. If code generation accelerates but test failures and rollback rates climb, the apparent efficiency is fake.

Do not let the model choose the metric. Let the business process choose it.

Measure transferred cost explicitly

Transferred cost is the hidden expense that appears when one function gets faster and another absorbs the cleanup. It is the most important thing to measure and the easiest thing to ignore. A CFO should want a ledger for it. A CTO should want instrumentation for it.

A simple transferred-cost ledger has four columns: where the AI was used, what local metric improved, where the extra work landed, and who paid for it. Example: support agents resolve cases 14% faster, but product managers now spend more time on unclear bug reports because the assistant encouraged overconfident escalation notes. That product-management time is not free. It belongs in the trial.

For engineering, the transferred-cost ledger should include code review burden, test maintenance, flaky-test repairs, incident follow-up, architecture cleanup, and documentation debt. For finance, it should include exception handling, audit follow-up, controls review, and policy clarification. For sales, it should include legal redlines, customer confusion, and CRM cleanup. In each case, the program’s apparent benefit should be discounted by the cost that moved elsewhere.

This is where many enterprise pilots break. They are measured at the point of model output, not at the point of business completion. The result is a local efficiency story that does not survive a full ledger.

Use one trial, not five disconnected pilots

Many organizations make the same mistake: they let each department run its own AI experiment, then compare anecdotal wins. That produces noise, not governance. The better approach is a single trial framework with shared definitions and local extensions. Define what counts as a task, what counts as completion, what counts as rework, and what counts as an exception. Then let each function add its own domain-specific measures.

A shared framework prevents the common CFO problem: one department reports labor savings while another reports quality loss, and nobody can reconcile the numbers. It also prevents the CTO problem: different teams evaluate different tool versions, prompts, policies, or approval rules, so the results cannot be compared.

The trial should include a no-AI baseline, an AI-assisted group, and a written policy for what the assistant is allowed to do. If the tool can draft but not send, say so. If it can summarize but not decide, say so. If it can propose code but not merge, say so. Boundary clarity is not bureaucracy. It is the only way to know whether the result is value or hidden risk.

What CFOs should ask before approving budget

CFOs should not ask whether AI is transformative in the abstract. They should ask five concrete questions:

1. What business task will change, and what is the current baseline?
2. Which downstream team absorbs the output, and how will their cost change?
3. What evidence shows the gain persists after review, rework, and exception handling?
4. What is the payback period under conservative assumptions?
5. What would make us stop or shrink the program?

If those questions cannot be answered, the project is still a demo. Budget should follow evidence, not optimism. CFOs should also insist on a transfer-cost reserve: if the pilot shifts work to another team, the receiving team’s cost must be counted, not treated as an externality.

A second useful CFO rule is to treat AI adoption like any other capital allocation decision. A tool that improves one workflow by 10% but increases downstream labor by 8% may still be worth it, but only if the downstream labor is cheaper or the quality gain matters. The decision should be made on unit economics, not enthusiasm.

What CTOs should enforce before rollout

CTOs should treat the model as one component in a controlled system. The system needs access control, prompt and policy versioning, audit logs, rollback paths, and clear ownership. They should also require repeatable evaluation on real tasks, not just benchmark theater.

At minimum, a CTO should demand these controls:

• versioned prompts, policies, and model settings
• clear permission boundaries for read, draft, and write actions
• logging for every material action and exception
• manual override and rollback for consequential outputs
• periodic re-evaluation against fresh task samples
• a way to disable the tool if quality drops or costs move elsewhere

CTOs should also be wary of overfitting the trial to the easiest users. The NBER support study found a much bigger gain for novice workers than for experienced ones, while METR found experienced developers were slower with AI in their trial. That is not a contradiction; it is a warning. Segment the workforce. Measure separately by seniority, task complexity, and domain risk. Averages hide the people who benefit and the people who pay the cost.

Evidence limits matter

The public studies are useful, but they do not justify blanket claims. The NBER support result comes from a customer support environment, not finance, legal, engineering, or operations. The BCG study measured consulting tasks under specific conditions. The METR study involved experienced open-source developers working on their own repositories. The MIT NANDA report is broad and influential, but it is not a universal law. Different tasks, incentives, and control structures produce different results.

That means the right interpretation is not “AI works” or “AI fails.” The right interpretation is: AI can create value when the workflow, incentives, and governance are aligned; otherwise it can shift cost, hide rework, and produce false confidence. A good trial is designed to reveal exactly which of those worlds you are in.

Decision makers should therefore avoid using vendor claims as a substitute for internal evidence. A pilot is not successful because the demo looked smooth or because a department liked the interface. It is successful when the organization can show that the same task got cheaper, safer, or faster end to end, with the transferred costs measured and acceptable.

A practical decision framework

If you want a simple gate, use this sequence:

1. Identify the workflow. Pick one high-volume task with a clear owner and clear downstream effect.
2. Set the baseline. Measure time, cost, quality, and exception handling before AI.
3. Define the boundary. Write down what the tool may draft, suggest, classify, or do on its own.
4. Run a controlled trial. Compare control and treatment groups under the same policy.
5. Track transfer. Measure where the extra work lands after the model output.
6. Review unit economics. Count labor, quality, risk, and customer impact together.
7. Decide with a stop rule. Expand only if the net effect survives conservative assumptions.

This framework is boring on purpose. Boring is what governance looks like when it is working. The point is not to prove that AI is impressive. The point is to prove that the organization is better off after the whole system absorbs the change.

That is the real CFO/CTO test. If the tool saves 10 minutes in one place and costs 12 in another, it is not a productivity tool; it is a transfer machine. If the tool saves 10 minutes, costs 2, and keeps quality steady or better, then you have something worth scaling. Everything else is theater.

Sources: NBER: Generative AI at Work; BCG/Harvard field study; METR developer trial; MIT NANDA report.