Using LLMs to Audit Whether Your Company's Initiatives Actually Worked

Hello. This is Keito Inoshita from Affectosphere Group.

Can your organization honestly answer the question “did this initiative actually work?”

You introduced a new training program. You changed your hiring criteria. You redesigned your ad creatives. Every time something gets launched, “impact measurement” comes up in conversation — but the number of organizations that actually conduct rigorous, reproducible evaluations is still small.

A study published on arXiv in June 2026 (Tobias Holtdirk et al.; arXiv:2606.13670) addresses this challenge from the academic side. The research team applied LLMs to automatically assess the reproducibility of 76 social and behavioral science papers, and found that LLMs matched or exceeded human re-analysts in accuracy.

This is not just about making academic research more efficient. The core idea translates directly to building internal evidence governance for organizational initiatives.

---

Three takeaways for today

1. LLMs matched or exceeded human re-analysts in assessing the reproducibility of social and behavioral science papers.
2. The underlying capability — automatically verifying whether an analysis can be reproduced from documented procedures — applies equally to internal initiative evaluations.
3. Business, HR, and marketing teams can draw concrete implementation ideas from this research for building a structured evidence governance system.

---

① Reproducibility isn’t just an academic problem

First, a quick definition of reproducibility.

Reproducibility means that if someone else follows the same procedure on the same data, they get the same results.

In academia, the “reproducibility crisis” has been a recognized problem for years. Landmark psychology experiments have failed to replicate when researchers tried to run them again.

But this problem exists inside companies too.

“I can’t figure out what assumptions the previous analyst used in this spreadsheet.” “Two people ran the same dataset and got different numbers.” “There’s a report showing the initiative worked, but no one can find the analysis methodology.” All of these are reproducibility problems.

What this research investigated is whether LLMs can automatically identify and verify reproducibility issues — without requiring a human re-analyst to go through the work manually.

---

② LLMs outperforming human re-analysts

The research evaluated 76 papers in the social and behavioral sciences.

The evaluation process involved taking each paper’s documented analytical procedure, running the analysis again, and checking whether the reported results could be reproduced.

The finding: LLMs performed at least as well as human re-analysts in determining whether results were reproducible.

This might be surprising. “Understanding and re-running academic analyses requires deep domain expertise, doesn’t it?” is a reasonable reaction.

But in practice, much of reproducibility assessment comes down to “follow the stated procedure, run the code, compare the output.” LLMs can combine code reading, execution, and output comparison to automate this workflow effectively.

Speed is also a factor. Human re-analysts can spend hours or days per paper. LLMs close that gap significantly.

---

③ Building an internal evidence governance system

Here is the practical question: how do you apply this idea to your organization’s initiative evaluations?

Let’s start by making the internal reproducibility problem concrete.

In corporate strategy, HR, and marketing teams, initiatives get evaluated all the time. Someone collects data using their preferred method, runs some analysis, puts together a slide deck, and presents the results. The conclusion — “the initiative worked” or “it didn’t” — gets documented.

The problem is that this analysis often lives only in one person’s head and their personal spreadsheet. The upstream assumptions, the preprocessing steps, the choice of comparison group — none of this is documented in a way someone else could verify.

Here is what an LLM-based evidence governance layer could look like:

The first component is automated documentation of analytical procedures. Rather than just producing a result, any initiative evaluation run with LLM assistance also generates a structured record of how the analysis was conducted — what data, what preprocessing, what comparison group, what statistical approach.

The second is automated reproducibility checking. Given that structured record, a different analyst or an LLM can re-run the same analysis and confirm whether the results match. Discrepancies trigger an alert for human review.

The third is evidence quality scoring. When an analysis claims “the initiative had a positive effect,” an LLM evaluates the strength of that evidence — checking sample size adequacy, comparison group design, time-series considerations, potential confounders.

---

④ Concrete use cases by team and KPI

Let me make this more specific for different business functions.

For corporate strategy teams, the application is pre-decision reproducibility review of initiative impact reports.

Before “this initiative worked, let’s expand it” gets incorporated into next year’s plan, an LLM verifies that the supporting analysis is documented in a reproducible form. Was the comparison group well-defined? Were external factors considered? The KPI to track: “percentage of initiative impact reports passing reproducibility review before inclusion in planning documents.”

For HR teams, the application is standardized evaluation of training and hiring program effectiveness.

When reporting on the impact of a new training program, an LLM reviews whether the analysis is reproducible — checking not just “scores improved pre vs. post” but “which group was the control, were external factors accounted for.” KPI: “rate of HR initiative evaluations submitted in standardized, reproducibility-auditable format.”

For marketing teams, the application is consistent cross-campaign comparison.

When comparing the effectiveness of multiple campaigns, inconsistent analytical assumptions between reports make honest comparison impossible. An LLM can review each campaign’s effectiveness analysis and flag assumption mismatches before the comparison is made. KPI: “rate of cross-campaign comparisons where analytical assumptions were confirmed consistent.”

---

Becoming an organization that can prove “it worked”

What this research demonstrated is that LLMs can take on the cognitive work of reproducibility evaluation — a task that previously required specialized human time.

The academic validation used 76 papers. Internal initiative evaluations are often simpler in structure, which means LLM-based reproducibility checking may actually be easier to implement than it was in the research setting.

Organizations that can back up “this initiative worked” with reproducible, verifiable evidence are stronger — both in terms of external accountability and in terms of internal decision quality.

LLM-powered evidence governance is still an underexplored area in most companies. Organizations that build this capability now will be ahead when data-driven decision standards tighten further.

That’s it for today!

---

Reference

1. Tobias Holtdirk et al. (2026). Automated Reproducibility Assessments in the Social and Behavioral Sciences Using Large Language Models. arXiv preprint arXiv:2606.13670.

* This article was written in part with AI assistance and may contain inaccuracies.