On this page
Part 5 of 9 AI training for employees
Previous: AI training ROI: what to measure, and what not to claimNext: AI adoption statistics: what the workplace data says
All parts
- How to train employees on AI: a 90-day plan for L&D
- AI training formats compared: video, workshop or hands-on practice online
- Best AI training for employees: 9 platforms compared by what learners actually do
- AI training ROI: what to measure, and what not to claim
- How to run an AI pilot program in eight weeks
- AI adoption statistics: what the workplace data says
- How to create training with AI: a five-step workflow for L&D
- Why we built Tokenacademy: AI is learned by doing
- AI literacy test for employees
The short answer: to run an AI pilot that ends in a decision, fix three things in the first week: the few tasks it covers, a baseline measured before anyone uses the tool, and the rule that says scale, adjust or stop. The rest of this article is how to get there in eight weeks with one cohort, and how to let that cohort practise on made-up data before it touches real data.
This is a guide to an AI pilot program, not to a company-wide plan. If you are building the larger programme around it, our guide to enterprise AI training covers that, and how to train employees on AI gives the 90-day plan. Here the question is narrower: how do you test an AI tool or use case with a small group, and learn something you can act on?
An AI pilot in eight weeks
- Weeks 1-2Scope3 to 5 tasks, baseline, decision rule
- Weeks 2-3PractiseMade-up data, in a sandbox
- Weeks 3-7Real workSame tasks, real data, weekly check-in
- Week 8DecideScale, adjust or stop, by the week-1 rule
Pilot, proof of concept or project: what is the difference?
The words overlap, so fix them for your team. A proof of concept asks whether something can work at all, usually with a few technical people. An AI pilot project asks whether it works for real people doing real work, with the training, the review steps and the managers they will have in production. This article is about the second.
The distinction matters because a pilot can succeed technically and still fail. The UK government's cross-government trial of Microsoft 365 Copilot gave licences to about 20,000 employees in 12 organisations for three months. Its headline was 26 minutes saved a day. The report itself says that the time savings are self-reported, and that 17% of users noticed no clear time saving. Treat that as the lesson on measurement: decide how you will measure before you start, not after.
Pick the tasks
Start with tasks, not tools. A pilot of "AI for the sales team" cannot fail or succeed; a pilot of "drafting first replies to inbound enquiries and summarising call notes" can.
Choose three to five tasks that meet four conditions:
- Frequent. People do them every week, so you see enough repetitions in a few weeks.
- Checkable. Someone can tell a good output from a bad one in a minute or two.
- Low stakes at first. An error is caught in review and does not reach a customer, a candidate or a regulator.
- Measurable now. You can time the task, count its errors or rate its quality today, without the tool.
Write each task as a sentence: who does it, what goes in, what comes out, what "good" looks like. Check that none of them needs data your data protection officer has not cleared for the tool you chose. If one does, it belongs in a later phase, or in the sandbox with invented data.
Pick the cohort
A cohort of 15 to 40 people in two or three related roles is enough to see patterns; this is a rule of thumb, not a statistical threshold. Include the people who will do the tasks and the managers who review the output.
Beware of a cohort made only of volunteers. In its evaluation of its own Copilot trial, the UK Department for Work and Pensions (DWP) gave 3,549 staff licences through volunteering and peer nomination, and compared them with colleagues without licences. The evaluation states that it was not a randomised controlled trial, that outcomes were self-reported and that there was no pre-trial baseline. It also warns that self-selection "may lead to an overestimation of Copilot's benefits". Volunteers are likely to be the people already interested in AI, so mix in some who are curious but sceptical.
Name one pilot lead who owns the calendar, the check-ins and the final report. Keep a comparison group if you can: colleagues doing the same tasks without the tool over the same weeks.
Train inside the pilot
A pilot plan often gives training one line: "kick-off session". That is a weak spot. If half the cohort has never written a request an agent can act on, you will be measuring their inexperience, not the tool's value. The DWP evaluation found a wish among its users for more targeted, role-specific training and quick-reference materials, and concluded that users may need targeted guidance and prompt libraries.
Build practice into the plan, in the order that protects you:
- Practise on made-up data first. Before anyone pastes a customer email or a CV into an agent, they rehearse the same kinds of tasks on invented material, in an environment where nothing real is at risk. That is what an AI sandbox is for, and AI sandbox training explains the method.
- Practise the tasks you picked. A generic course teaches generic habits. Your cohort should rehearse the three to five pilot tasks. Video, workshop or hands-on formats compared shows what each can rehearse.
- Teach checking, not only asking. The habit that protects you is reading the output critically before it goes anywhere.
Practice first, real data second
Weeks 2-3: made-up data
- Learn what to ask and check
- Mistakes cost nothing
- Same tasks as the pilot
Weeks 3-7: real data
- Apply it to real work
- Keep a human review
- Log what went wrong
This is where Tokenacademy fits, as one way to do it. Each employee practises with a real AI agent on a real project, in the browser: there is nothing to install, no AI account to create and no licence to buy per learner. Learners connect the agent to simulated company tools on a made-up company, read only, so the skills are real and no real data is ever at risk. Sessions last about 45 minutes and their milestones are checked automatically, so a pilot lead does not have to run a classroom. To see how that would map onto your tasks, book a demo.
One card shows a session's level and duration, the other a progress bar for a session already started
Measure and decide
Pick one primary measure and two or three supporting ones. For each, record the baseline before week 3.
| Measure | How | Watch out for |
|---|---|---|
| Time per task | Time a sample, with and without the tool | Self-reports run optimistic |
| Quality | A reviewer scores blind, on a fixed rubric | Reviewer drift over weeks |
| Rework | Share of outputs sent back | Hidden when nobody logs it |
| Adoption | Who still uses it in week 6 | Novelty fades |
Quality deserves as much weight as speed. The DWP evaluation asked about work quality as well as time, and it describes Copilot as a complement to human expertise, not a replacement. A pilot that saves minutes and adds rework has not saved anything. For reporting results to finance without overclaiming, use the indicators and the one-page template there.
Then write the decision rule in week 1, while nobody is attached to the result.
Write the decision rule in week 1
Scale
- Measure beat baseline by your margin
- Cohort and managers want to continue
Adjust
- Some tasks passed, others did not
- Rerun shorter, on tasks that passed
Stop
- No measure moved, or quality fell
- Write down why, and what retry needs
If you want the wider evidence on how companies and employees are adopting AI, see AI adoption statistics. If your pilot touches staff in the EU, the AI Act's AI literacy requirement is covered in Article 4 and AI literacy.
How to run an AI pilot in eight weeks, week by week
- Weeks 1-2: scope. Choose the tasks and the cohort, agree the measures, take baselines, write the decision rule, clear the data questions.
- Weeks 2-3: practise. The cohort rehearses the pilot tasks on made-up data, in short sessions.
- Weeks 3-7: real work. The same tasks on real data, with human review. A 30-minute check-in each week; the pilot lead logs what went wrong.
- Week 6: mid-point check. Look at adoption and rework. Fix training gaps now, not at the end.
- Week 8: decide. Compare the measures with the baseline, apply the rule, write one page: decision, evidence, what you would change.
FAQ
What is an AI pilot program?
A time-boxed test of an AI tool or use case with a small group doing real work, with measures set in advance and a decision at the end. It differs from a proof of concept, which tests whether something can work at all.
How long should an AI pilot run?
Eight weeks is a workable length: two for scope, one or two for practice, then enough weeks of real work to see habits settle. A much shorter pilot may only capture the first excitement. This is our judgement, not a figure from a study.
How many people should be in an AI pilot?
Enough to see patterns across roles and still manage closely, which for most teams means roughly 15 to 40 people. That range is our judgement, not a figure from a study. Add a comparison group of colleagues doing the same tasks without the tool if you can.
What if the pilot fails?
A pilot that ends in "stop" with a written reason has done its job: you spent weeks and a small cohort's time rather than a full rollout. Record what you would need to see to try again.
Should the cohort use real data from day one?
We would not. Let people rehearse the pilot tasks on made-up data in a sandbox first, then move to real data with review. It lets you separate "they did not know how" from "the tool did not work".
Sources
- UK Government, Microsoft 365 Copilot experiment: cross-government findings report: 20,000 licences, 12 organisations, three months, self-reported time savings, 17% with no clear saving. Checked 15 November 2026.
- UK Department for Work and Pensions, An evaluation of DWP's Microsoft 365 Copilot trial: 3,549 licences, non-randomised comparison, self-selection caveat, wish for role-specific training. Checked 15 November 2026.
