How to evaluate agency case studies without believing the brochure

Start with seven fields

Tim's 7-Minute Case Study encourages agencies to connect a client's problem, the work and the result in a concise story. For buyers, that compact story should lead to a deeper discussion of evidence and relevance. Brevity is useful; it cannot replace the detail needed for a hiring decision.

Ask what the number measures

Revenue, attributed revenue, contribution, conversion rate and customer count describe different things. A case study should use a clear label and retain it through the explanation.

A reported 50% increase might be a change from two sales to three, or from 20,000 to 30,000. Both use the same percentage; the evidence available and operating implications differ considerably. Ask for absolute values or a meaningful range if exact figures are confidential.

Distinguish percentage points from relative change. Moving conversion from 2% to 3% is a one-percentage-point increase and a 50% relative increase. That mathematical result says nothing by itself about traffic quality, profit or causality.

Check the denominator. A growing count of repeat customers can coexist with a flat repeat-purchase rate if the total customer base is also growing.

A real example of a useful distinction

Klaviyo's Half Magic case study reports fivefold year-over-year growth in the number of repeat purchasers over its stated period. It names Lilo Social and describes changes to the retention setup. The claim concerns a customer count, so it should not be rewritten as a fivefold improvement in retention rate. Half Magic case study.

The buyer's follow-up should ask about acquisition growth, cohort sizes, customer mix and the baseline. Those questions do not establish that the result is wrong. They establish what would be needed to interpret it accurately. The source is a platform vendor's marketing case, rather than an independent experiment.

Compare the time periods fairly

Ask whether the result compares equivalent seasons, campaign lengths and stages of the customer lifecycle. A promotional week compared with an ordinary week can produce an accurate number and a misleading impression.

Find out whether prices, budgets, inventory, distribution or the product itself changed. Check for tracking changes too. A repaired conversion event can increase reported conversions without an equivalent increase in actual sales.

For B2B, ask whether lead and revenue cohorts had comparable time to mature. For ecommerce, understand how returns and cancellations were handled and whether repeat purchases were observed over a consistent window.

Test the economics with a worked example

Consider this fictional before-and-after case. Revenue is net of returns and discounts. The contribution margin assumption is after relevant variable costs and before advertising and agency fees. Assume all revenue is included consistently in both periods.

The revenue grew 60%, while the illustrated post-marketing contribution stayed flat. This does not automatically mean the work was unsuccessful. The business might value additional customers or expect defensible future purchases. It does mean that the revenue headline cannot establish increased current profit.

The example also does not establish causation. You would still need to know what would likely have happened without the changes.

Separate contribution from causal proof

A plausible explanation identifies how the agency's work affected the customer's behavior or the business's operation. Stronger causal evidence may involve a well-designed experiment or another appropriate evaluation method. A before-and-after story alone leaves room for other explanations.

Ask the agency which evidence supports its role and which uncertainty remains. A thoughtful answer might acknowledge a simultaneous pricing change or the contribution of a strong internal team. Taking credit for every improvement is a weak substitute for explaining the work.

U.S. FTC guidance also makes clear that endorsements do not eliminate the need to substantiate advertising claims. A testimonial should not be treated as independent proof of the underlying performance claim. FTC endorsement guidance.

Match the starting conditions to your business

Compare customer economics, budget, internal resources, buying cycle and implementation capacity. Ask whether the people responsible for the case will work on your account and whether the method still applies under current platform conditions.

A large brand may have supplied research, creative and engineering resources your company does not have. A case built around unusually generous discounting may be irrelevant to your margin constraints. An impressive result can be genuine and still be poor evidence for your decision.

Verify without demanding another client's secrets

Ask for a reference, a redacted report or a private walkthrough under appropriate confidentiality. Respect limits on client data. An agency should be able to explain its method and role without handing over raw personal data or commercially sensitive records.

Record unresolved claims as unverified. Lack of public detail is not proof of failure, but it limits how much weight you can put on the story. A logo alone should not become proof of current work, scope or results.

Use the buyer's scorecard to compare the evidence across finalists. The strongest case study for your search is the one that helps you understand what the agency can do under conditions like yours, including what it cannot promise.