When Per Scholas rolled out an AI-powered intelligent tutoring system two and a half years ago, it found that learners who used the tool were more likely to pass their industry certifications. There was just one problem: nearly half the learners never used it. 

This was one of the most valuable insights from five years of rapid-cycle learning by Per Scholas and its learning partner, the American Institutes for Research (AIR), and it changed how we approach AI initiatives.

Workforce organizations are under enormous pressure to integrate AI into both their training and operations. New AI-powered tools are hitting the market every day. The temptation is to move fast, run pilots, and sprint to the next tool. But after years of doing this work together, we’ve learned that getting AI right means occasionally slowing down.

Breaking Out of the Pilot Trap

Most workforce organizations are familiar with this cycle: launch a pilot, gather some data, write a report, move on to the next thing. We call it the hamster wheel. Each pilot generates insights, but nothing builds on what came before.

What we’ve tried to build instead is a system in which each round of testing feeds directly into the next, metrics carry over, and the organization gets measurably better at adopting new tools over time. It sounds a lot easier than it turned out to be, but we’ve learned lessons over the years that can guide others doing similar work. 

Start With Why It Should Work

Before testing any AI tool, we map out exactly how we expect it to create value. What’s the theory of change? What assumptions are we making about how learners or staff will actually use it? For a training provider like Per Scholas, having an external learning partner like AIR to pressure-test those assumptions has been essential. 

The people closest to the work bring deep expertise, but proximity can also create blind spots. Those early conversations have caught flawed assumptions before they became expensive mistakes.

Measure What Actually Moves

For workforce organizations, the outcomes that matter most, such as job placement, can take months to materialize. You can’t wait that long to decide whether a tool is working. So we’ve gotten deliberate about identifying earlier signals that predict whether we’re on the right track. That means defining metrics before launch and being honest about what the data is actually telling us.

We also try to resist the urge for perfection. We start with what we can collect without overhauling existing systems, log the gaps we can’t fill yet, and layer in richer data, including interviews, observations, and pulse surveys, through the learning partnership.

Adoption Is the Bottleneck

Getting the rollout right is just as important as getting the technology right. With the tutoring system, the tool worked when people used it. The problem wasn’t resistance; it was that learners preferred the tools they already knew. Now, we treat adoption itself as a design challenge, testing different onboarding strategies and support models before assuming any tool will gain traction on its own.

“Build it and they will come” is a common trap in social impact work, and it’s just as common with AI tools. 

Culture Over Tools

None of this works without an organizational culture that supports it. At Per Scholas, a relentless focus on “jobs, jobs, jobs” has provided clarity that makes it easier to evaluate whether an AI initiative is worth pursuing. It also means AI adoption is woven into everything staff does, because that’s the only way to deliver the skills that employers want at scale and at the pace of rapidly changing technology. 

On the research side, AIR has had to evolve. Traditional research timelines don’t match the pace of AI-driven program changes. Being a useful learning partner means staying close to implementation, adapting study designs in real time, and being willing to answer messy questions with the best available evidence.

The most important factor is how an organization treats findings it doesn’t like. Our tutoring tool could have been filed as a disappointment. Instead, we’re using the adoption data point to improve our AI implementation strategy. Honest results need to be seen as assets, not threats.

What Comes Next

We’re now applying these lessons to AI-powered career coaching and job placement tools, building on those adoption insights. The goal isn’t just to test whether AI works in workforce training; it’s to build an organization capable of continually adapting as the technology evolves.

For workforce organizations on their own AI implementation journeys, our main advice is this: don’t wait for the perfect tool or the perfect study. Start with a clear theory of change, measure what you can, take adoption seriously, and create the conditions for honest learning. The organizations that build this capacity now will be far better positioned as AI reshapes the labor market, not because they picked the right tool first, but because they learned how to keep picking better ones.

Samia Amin is a managing director at the American Institutes for Research. Tamara Johnson is the chief operating officer at Per Scholas.