The Method for Measuring AI Feature ROI

AI features are easy to ship and hard to evaluate. Cost is visible in the monthly token bill; value is diffused across user behaviors that may or may not be caused by the feature. Most teams declare success on usage rate and never look again. This article proposes a three-layer measurement framework and names the kill criteria for features not earning their token spend.

1. The claim: most AI feature ROI is measured wrong

The conventional metric for AI feature success is engagement: "X% of users tried the AI feature, Y% used it more than once, Z% have used it in the last 30 days." These numbers are easy to compute, look good in dashboards, and tell you nothing about whether the feature is worth what it costs.

Three failure modes of usage-only measurement:

• Self-selection bias. Users most likely to engage with new AI features are users most likely to be happy with the product anyway. Usage correlates with retention because both correlate with engagement, not because AI causes retention.
• Cost blindness. A feature with 80% usage and $4/user/month token cost on a $20/user/month subscription consumes 20% of gross margin. Usage-only measurement does not surface this.
• Lock-in to bad features. 80% usage cannot be killed because "users love it." But love is not the same as paying. If users would pay without the feature, it is decorative cost.

MIT Sloan Management Review's research on AI ROI^[2] consistently finds that organizations measuring AI by usage alone over-invest in unjustified features and under-invest in features that quietly drive retention. Harvard Business Review's coverage of AI product ROI^[1] reaches similar conclusions: usage is a vanity metric; cohort retention and revenue attribution are the metrics that decide whether a feature pays for itself.

2. The three layers of AI feature ROI

The correct method has three layers, each answering a separate question:

1. Layer 1: marginal cost per invocation. How much does the AI feature cost to run, per active user, per month? Inputs: token cost, embedding cost, vector storage, additional infrastructure. Outputs: dollars per active user per month.
2. Layer 2: marginal revenue attribution. How much revenue is attributable to the AI feature specifically? Inputs: cohort comparison between AI-feature users and matched non-users. Outputs: revenue uplift per AI-feature user per month.
3. Layer 3: retention and expansion impact. How does the AI feature affect 30/60/90-day retention and expansion? Inputs: cohort retention curves, expansion revenue rates. Outputs: retention uplift, expansion uplift.

Net AI feature contribution per active user per month = Layer 2 + Layer 3 - Layer 1. If positive, the feature is paying for itself. If negative, the feature is a cost center.

Each layer requires different data and patience. Layer 1 is hours of work from the token billing dashboard. Layer 2 requires 30-60 days of cohort data. Layer 3 requires 90+ days. Most teams stop at Layer 1 and skip Layers 2 and 3 because they require patience.

3. Layer 1: marginal cost per AI invocation

Layer 1 is the easiest to measure and the most-often correctly measured. The formula:

Layer 1 cost per active user per month = total monthly AI spend / monthly active users of the AI feature

Components of total AI spend:

• Token cost. Input + output tokens × per-token pricing. Anthropic Claude^[6], OpenAI, and Google all publish per-token rates. Multiply by monthly volume.
• Embedding cost. For RAG features, the embedding API spend.
• Vector storage cost. Pinecone, pgvector, or equivalent monthly subscription.
• Additional infrastructure. Any incremental compute, storage, or networking attributable to the AI feature specifically.

A typical Layer 1 calculation for an AI-summarization feature on a $40/mo SaaS: $400/month total AI spend, 200 monthly active users of the feature → $2/user/month Layer 1 cost. The feature costs 5% of the subscription price per user. This is the marginal cost — the threshold the feature must clear in Layers 2 and 3 to be net-positive.

4. Layer 2: revenue attribution to AI feature

Layer 2 is the layer most teams skip because it requires cohort comparison. The question: do users who use the AI feature pay more (via plan upgrades or usage-based billing) than matched users who do not?

The right counterfactual: matched-cohort comparison. Find users who use the AI feature (treatment cohort) and users who match on observable characteristics but do not use the feature (control cohort). Compare monthly revenue per user across cohorts after 30, 60, 90 days. The difference is Layer 2 revenue attribution.

Matching criteria for the cohorts:

• Same plan tier at start of period
• Same usage volume on non-AI features
• Same tenure (months since signup)
• Same industry or segment if applicable
• Similar engagement scores on non-AI features

The matching does not need to be perfect — even crude matching produces directional Layer 2 numbers. The AI Feature Attribution calculator^[7] automates the cohort comparison for solo founders without dedicated analytics infrastructure.

A typical Layer 2 result for an AI-summarization feature: treatment cohort generates $42/user/month (slight upgrade behavior), control cohort generates $38/user/month → Layer 2 revenue uplift = $4/user/month. Combined with Layer 1 cost of $2/user/month, the feature is net-positive on Layer 1+2 alone ($2 net contribution per user per month).

5. Layer 3: retention and expansion impact

Layer 3 is the layer that distinguishes good AI features from great ones. The question: do users who engage with the AI feature retain at higher rates and expand to higher tiers more often than matched non-users?

ChartMogul's 2024 SaaS Retention Report^[5] documents that feature-engagement gates are the strongest controllable predictor of cohort retention in self-serve SaaS. A user who engages with three or more features in the first 30 days retains at 1.5x to 2x the rate of users who engage with one or fewer. AI features tend to be high-engagement gates if positioned correctly.

Layer 3 computation:

• 30-day retention uplift. What percentage of treatment-cohort users are still subscribed at day 30 vs control cohort? Convert to dollars by multiplying retention difference by monthly ARPU and customer lifetime.
• 60-day retention uplift. Same calculation at 60 days. Captures longer-tail retention effects.
• 90-day expansion uplift. What percentage of treatment-cohort users upgraded plans, added seats, or increased usage-based billing in the first 90 days vs control? Convert to dollars by expansion-revenue-per-user.

A typical Layer 3 result for an AI-summarization feature: 5% retention uplift at 60 days (75% retained vs 70% for control), 12% expansion rate uplift at 90 days (15% expanded vs 13.4% for control). At $40/mo ARPU and 18-month customer lifespan, this translates to roughly $3-$6/user/month of additional Layer 3 contribution.

Total net AI feature contribution in this case: Layer 2 ($4/user/month) + Layer 3 ($3-$6/user/month) - Layer 1 ($2/user/month) = $5-$8/user/month net. At 200 monthly active users, the feature is contributing $1,000-$1,600/month of net value. Defensibly positive.

6. The trap: counting headline metrics instead of marginal contribution

The most common error is reporting Layer 2 and Layer 3 numbers as the treatment cohort's absolute values rather than the difference vs control. "AI feature users have 75% 60-day retention" sounds good. Without context, it is meaningless — the control might also have 75% retention, in which case the AI feature is contributing zero retention.

Three forms of this trap:

• Absolute retention reporting. "Users who use the AI feature retain at 75%." Compared to what? Without the control, the number is decoration.
• Survivor-bias attribution. "Users who used the AI feature 10+ times have 90% retention." Yes, because they were already engaged. The AI feature did not cause the engagement; the engagement caused the AI feature use.
• Aggregated revenue framing. "AI feature users generate $200/month of ARR." If the control cohort generates $195/month, the feature contributes $5/user. The headline of $200 is misleading.

The discipline: always report Layer 2 and Layer 3 as delta vs control cohort, never as absolute values. The Bessemer State of the Cloud 2024^[3] framework for AI-native SaaS metrics consistently emphasizes delta reporting as the only defensible attribution standard.

7. Worked case: AI-summary feature on a $40/mo SaaS

Concrete walk-through. A document SaaS at $40/mo subscription, 1,000 paying customers, ships an AI-summarization feature that condenses long documents into bullet summaries.

Layer 1 (marginal cost): 200 monthly active users of the feature, generating ~50 summaries/month each at 2,000 input tokens + 400 output tokens per summary on Claude Sonnet 3.5. Token cost: 200 users × 50 summaries × 2,000 input tokens × $3/M = $60/month input. 200 × 50 × 400 output × $15/M = $60/month output. Total ~$120/month, or $0.60/user/month. Plus eval/monitoring overhead ~$80/month. Layer 1 = $200/month total, $1/user/month.

Layer 2 (revenue attribution): Cohort comparison after 60 days. Treatment cohort (200 AI users): $41/user/month average revenue (some upgrades to higher-volume tiers). Control cohort (matched 200 non-AI users): $39.50/user/month average. Layer 2 uplift = $1.50/user/month.

Layer 3 (retention/expansion): 60-day retention: 78% treatment vs 73% control (5pt uplift). At $40 ARPU and 18-month lifespan, retained-customer LTV is $720; 5pt uplift on 200 users = 10 additional retained users × $720 = $7,200 of additional LTV over the next 18 months, or $400/month amortized. Plus 90-day expansion: 14% treatment vs 11% control (3pt uplift), times average expansion value $80/year × 200 users × 3% = ~$40/month. Layer 3 = $440/month total, $2.20/user/month.

Net contribution: Layer 2 ($1.50) + Layer 3 ($2.20) - Layer 1 ($1.00) = $2.70/user/month × 200 users = $540/month of net positive contribution. At the $40 subscription price, the feature is contributing roughly 7% of net margin against subscription gross. Worth keeping; worth investing in.

8. Objections and edge cases

"My SaaS does not have enough users to do cohort comparison." True under 200-300 monthly active users; statistical noise dominates the signal. The pragmatic alternative is qualitative measurement (interview 5-10 users who use the feature about why they pay) plus Layer 1 monitoring (verify the feature does not lose money on tokens). Move to quantitative measurement once user count supports it.

"My AI feature is free for users — Layer 2 attribution is zero by definition." The attribution to a free feature is via retention and expansion, not direct revenue. Run Layer 3 alone. Many AI features are positioned as plan-tier differentiators (free on Pro, not on Basic); attribution is then about upgrade behavior, which is Layer 3 expansion.

"The feature is strategic — we keep it for positioning even if ROI is negative." Sometimes valid. Some features are positioning anchors (a checkbox on the marketing page) rather than revenue drivers. The discipline is to be explicit: this feature is strategic, costs $X/month, and is funded as a marketing expense, not a product investment. Without that explicit framing, strategic features quietly compound cost.

"I cannot afford to do this analysis for every feature." Then do it for the top three by cost. The 80/20 rule applies: a few features account for most of the AI spend. Focus measurement effort on the expensive features; cheap features get Layer 1 verification only.

9. Implementation: the monthly AI feature review

The monthly AI feature review template, 60-90 minutes per month:

1. Pull total AI spend from billing. Anthropic, OpenAI, Google, plus any infrastructure (Pinecone, etc).
2. Allocate spend per AI feature. If you have multiple AI features, attribute the spend by usage. Most analytics platforms support this.
3. Compute Layer 1 per feature. Spend per active user per month.
4. Pull cohort retention and expansion data. Most billing platforms (Stripe, Paddle^[4]) export this directly; analytics platforms (Mixpanel, Amplitude) compute it from event data.
5. Compute Layers 2 and 3 deltas vs control cohort. Use the AI Feature Attribution calculator to automate the math.
6. Compute net contribution per feature. Layer 2 + Layer 3 - Layer 1. Positive = feature pays for itself. Negative = feature is on review.
7. Track over time. Net contribution should be stable or improving. Declining net contribution is a kill signal.

10. When to kill an AI feature

Three conditions for killing a feature:

1. Net contribution is negative for 3+ consecutive months. The feature is costing more than it earns and the trend is not improving.
2. You have already tried at least one optimization. Prompt shortening, cheaper model variant, caching, or removing features that drive cost without driving value. If optimization did not move the number, the feature is structurally underwater.
3. The feature is not strategic. If you have explicitly classified the feature as strategic (positioning, marketing anchor), the kill criteria are different. Strategic features are killed when the strategic value is gone, not when the ROI is negative.

Kill is the right answer roughly 30% of the time for shipped AI features at solo-founder scale. Most teams kill at 0% — features accumulate forever, the AI bill grows, and individual features cannot be justified in isolation. The discipline of three-layer measurement makes kill decisions possible because the data is clear; without the measurement, every feature is sacred because nobody can say what it contributes. The 2026 AI solopreneur stack covers the broader infrastructure and the retention playbook covers the Layer 3 retention work in detail.

11. FAQ

How do I measure AI feature ROI? Three layers: marginal cost (L1), revenue attribution via cohort comparison (L2), retention/expansion impact (L3). Net = L2 + L3 - L1.

Why is usage not enough? Usage measures interest, not value. 80% usage with zero revenue lift means the users would stay anyway.

How small before measurement is not worth it? Under $1/user/month cost, verify L1 only. Above $5/user/month, full three-layer required.

When should I kill? Net contribution negative for 3+ months after at least one optimization attempt. Kill is the right answer roughly 30% of shipped AI features.