AI literacy under Article 4 is an adoption problem

Since July 2026 the AI Act asks providers and deployers to take measures that support their staff's AI literacy, without requiring any specific level from any individual. I learned how that kind of duty plays out from analytics adoption at JD Sports Group, where getting people trained was the easy part and keeping the habit was the real work.

What changed in Article 4

Article 4 has applied since 2 February 2025. The original text asked providers and deployers to take measures to ensure, to their best extent, a sufficient level of AI literacy among their staff. The Digital Omnibus on AI (Regulation (EU) 2026/1744, in force since 27 July 2026) rewrote it. Providers and deployers now have to take measures to support the development of AI literacy of their staff, the text adds that this does not require any specific level of AI literacy of any individual, and the Commission has to publish practical examples of how to comply (AI Act Explorer).

The Commission's AI literacy Q&A says there is no need for a certificate, that relying on the instructions for use is not sufficient, and that organisations can keep an internal record of their training and guidance.

Read quickly, the new wording sounds lighter. In practice it leaves each organisation to decide what support means for its people and how it will know whether the support is working. I had to answer the same question for analytics.

What analytics adoption taught me

At JD Sports Group I was product owner of a programme of 8 measurement projects. The result I value most from it is a measure of behaviour: adoption of analytics grew 70% year on year. Stakeholder satisfaction was 90%, which was good to hear, but adoption told us whether people used the data when they made decisions.

Two practices were part of the programme from the start. Each implementation came with an operational dashboard, so people learned on the tool they would use. When an implementation moved into business as usual, a short survey checked whether it met the need it was built for.

The checkout project taught me the other half. Customer Care logged the reason for each live chat in Salesforce, and the weekly report built on those reasons was still used in their meetings long after the project closed. As the team hired new people, the logging discipline dropped and the reasons became less reliable, while the report kept arriving every week. The plan had been a one-off handover, so I made periodic retraining of Customer Care part of how the report was maintained.

Why AI tools behave the same way

An AI tool launched with one training session is in the same position as that report. People who join after the session never had it, the tool changes with each model update, and the habits that make output safe to use fade quietly: checking a claim against its source, or knowing which customer data must never go into a prompt. Nothing breaks on the day the habit goes, which is why a course completion rate says little about it.

Day to day I use AI to automate repetitive tasks, and my own time goes on verifying the output before I sign it off. That verification step is the first thing I would teach.

What I would do today

1. Start with an inventory of who uses which AI tools and for what. The role decides what literacy means: someone drafting customer replies with an assistant needs different knowledge from an analyst building a feature on a large language model.
2. Put AI guidance into onboarding, and repeat it on a date in the calendar and whenever a tool or its use changes.
3. Measure what people do with the tools. Two useful signals are the share of AI-assisted outputs that someone reviewed before they went out, and the incidents and near misses that people report.
4. Keep a dated record of the support each group received. The Q&A asks for no certificate, and a record is the simplest way to show what was done and when.

Writing analytics specs that developers can build

In UK government digital teams I wrote the specifications for my own team's analytics work, as user stories with acceptance criteria. The rule I took from it is easy to state: an analytics request is ready for development when someone other than the analyst can test it.

The situation

I worked as an analyst in UK government digital teams, at GDS and the Department for Education, often in several teams at the same time. The teams were multidisciplinary and worked in Agile, with product and delivery managers, developers, designers and user researchers on the same service.

Part of the work was digital identity. In the UK that is harder than in many countries, because there is no national identity card to check people against, so a service has to build confidence in who someone is from several weaker pieces of evidence. Government is still working on it: in February 2026 HMRC started signing up new customers through GOV.UK One Login, which GOV.UK said was then available for over 200 government services.

In a team like that, an analytics request competes with everything else on the board. If it arrives as "can we track the identity check?", the developer has to guess what to send, when and with which values. The guesses come back at user acceptance testing (UAT), when changing them is expensive.

What I did

For my own team, I wrote each analytics task as a user story with acceptance criteria. The story said who needed the data and which decision it supported. The criteria said what the developer had to build and how we would check it in UAT.

The example below is invented for this post and has no detail from a real service, but it has the shape I used.

As a performance analyst, I want an event when a user leaves the identity check step, so that we can report completion by step and see where people drop out.

• When the user leaves the step, one event named identity_check_step_end is sent.
• The event has an outcome parameter with one of these values: passed, failed, exited.
• The event sends no name, date of birth, document number or free text.
• In UAT, one attempt with each outcome produces exactly one event per attempt in the test property.

The privacy line sat in the criteria on purpose. In an identity service the safest personal data is the data the event never carries, and a written criterion can be tested in UAT like any other.

Writing the check first also moves the questions earlier. Edge cases come up while the story is being refined, when an answer is a comment on the ticket.

The team also automated its recurring reporting with Python and ETL pipelines. That cut the analytics effort on reports by around three quarters, and weekly reports went out on time to more than 30 stakeholders. It was a team achievement, built in Agile with everyone on the team.

What I learned

Acceptance criteria that cannot fail are a wish list. Each line has to describe something a tester can see happen or not happen. The analyst is the best person to write those lines, because the analyst knows what the data will be used for and which mistakes would make it useless.

What I would do today

The GOV.UK Service Manual already asks teams to have a performance analyst from the start of discovery and to work out their measurements while the service is built (Service Manual). I would keep that and add four things.

1. Link the tracking plan to the stories, so each event points to the story that created it and to the metric it feeds.
2. Check event names and allowed values automatically in the test environment, so a typo fails before release.
3. Agree once with the privacy team a list of fields that never go into analytics, and paste it into every story.
4. Write acceptance criteria for AI features in the same way. A feature built on a large language model also needs written cases it must pass before release, and someone other than its builder should be able to run them.

Agree the metric before you build the dashboard

At JD Sports Group, marketing and ecommerce each had their own figures and neither trusted the other's. The programme that fixed it started with definitions agreed by the managers, and the dashboards came afterwards.

The situation

From 2021 to 2024 I was Senior Digital & Marketing Data Analyst at JD Sports Group and product owner of a programme of 8 projects to improve measurement end to end. When the programme started, each manager had their own objectives and their own KPIs. Marketing did not trust the ecommerce figures, ecommerce did not trust the marketing ones, and meetings that should have ended in a decision ended in a discussion about whose report was right.

The request that usually arrives in that situation is a new dashboard. Built on top of the disagreement, it would have shown both sets of numbers on one screen, and the discussion would have carried on in front of it.

What I did

I started with a measurement plan, before choosing any tool. I ran sessions with the managers to agree the business objectives first, and then the KPIs that would show whether we were meeting them, for internal users and for customers.

To give it a structure I used the GDS Performance Framework, which separates primary and secondary metrics. Each area ended up with a small number of metrics that mattered, each with one shared definition behind it. GOV.UK has since withdrawn that framework page and points readers to its Service Manual guide on measuring success, which asks teams to work out their measurements while they build the service.

I split the roadmap in two. The strategic part was objectives, metric definitions and ownership. The operational part was instrumenting the tracking with developers and building dashboards. We ran it as an agile programme, with a Trello board for the programme and Jira for the delivery work with developers. New requests arrived every week, so the backlog kept moving, and I prioritised by dependency and by how many teams a piece of work would unblock. A metric was defined before it was implemented.

Dashboards were released one implementation at a time. After each one, people could see an operational dashboard, use it and tell us what was missing, and most of the refinement of the metrics happened at that stage. When a new implementation moved into business as usual, a short survey checked whether it met the need it had been built for.

The programme delivered its 8 projects with 90% stakeholder satisfaction, and adoption of analytics grew 70% year on year. Adoption is the figure I value most, because it shows that people started coming to the data to settle questions.

What I learned

Implementing a metric that nobody has agreed on moves the argument downstream. In a meeting the disagreement is visible and cheap to resolve. Inside a dashboard it looks like a data quality problem, and tracing it back to a definition takes much longer.

The second lesson came from releasing early. A definition agreed in a workshop is still a draft until people use it with real data, and the first dashboard was the cheapest place to find out what the draft had missed.

What I would do today

Creating a metric keeps getting easier. With Tag Manager templates and point-and-click tagging, a new event can be live in minutes. When an event takes minutes to create, the definition is the part of the work that needs an owner.

If I ran the same programme now, I would add four habits.

1. A written definition for every primary metric, with the business question it answers, the formula, the data source, the filters, the owner and the date it was agreed.
2. A change log that the dashboard shows. When a definition changes, the date appears on the chart, so nobody reads a step in the line as a change in the business.
3. No new event in the tracking plan unless it feeds a metric that has a definition.
4. A review of the secondary metrics every quarter, to retire the ones nobody opens.

Before your next dashboard

Take the headline metric of the dashboard you are about to build and ask two managers who use it to write down, separately, how it is calculated. If the answers differ, that conversation comes first, and the dashboard can wait a week.

A hole in the checkout funnel: fix the tracking before you redesign the page

At JD Sports Group the business was ready to redesign the checkout because the funnel looked broken. The first problem I found was a page nobody had tracked, and fixing it changed what the rest of the project could do.

The situation

I was Senior Digital & Marketing Data Analyst at JD Sports Group from 2021 to 2024. One project started with a clear request: the checkout funnel is broken and needs redesigning. Several teams had a view on what was wrong and what to change, and the pressure was to start changing things.

What I did

I started with the analysis. Compared with benchmark, the funnel numbers were off in a way that did not fit a checkout that had not changed recently. Following the drop led me to a page that had been added to the checkout at some point and never tracked. Customers were passing through a step the funnel could not see, which left a hole in the report.

We fixed the tracking first. That gave the teams a funnel they could trust, and it gave me the credibility to ask for more advanced tracking, which was approved.

The numbers then showed where people left the checkout, but they could not say why. I added a survey with Hotjar, the tool the team already had, and used the OpenAI API to clean and categorise the open-text answers. Payment problems came out as the main theme, backed by what customers had written.

Session recordings came next, but the volume was too high to turn into a priority list. So I changed approach and worked with Customer Care to put live chat on the checkout during working hours, on one condition: agents logged the reason for each conversation in Salesforce. That gave us the reasons directly from customers, every week.

There were competing requests. Marketing wanted Customer Care to run video calls with customers, and Customer Care wanted the chat on product pages. I argued for the checkout first by walking both teams through the funnel step by step, showing what data we already had at each stage and what each option would tell us that we did not know yet. The reporting stayed simple, with charts and a weekly report, and the teams started reusing it in their own meetings. Once the report was useful to them, the priority was no longer up for debate.

The result, and how it was measured

Checkout conversion improved by 4% (relative) after the changes. That figure comes from a before-and-after comparison against the baseline before launch. There was no A/B test, so the comparison cannot separate the effect of the changes from seasonality or from anything else that moved in the same period. When I present the number, that limit goes next to it.

The weekly report outlived the project, and Customer Care was still using it in their own meetings long after it closed. It also showed me the weak point. As Customer Care hired new people, the logging discipline dropped and the reason data became less reliable, so the reports kept coming out while their value went down. Retraining became a recurring task, built into how the report was maintained.

What I learned

A funnel is a data product, and the cheapest data quality control is checking that every step in it is measured before anyone draws conclusions from it. The epilogue taught me the second lesson: data that depends on people typing a reason into a CRM needs the same upkeep as a pipeline, because adoption work does not end at handover.

What I would do today

I would add four things.

• Tracking in the definition of done. A new checkout step does not go live until its event is in the tracking plan and a test confirms that the event arrives.
• An automated completeness check on the funnel. Pages inside the checkout path that receive sessions and send no funnel event should raise an alert.
• The LLM categorisation treated as a measurement instrument. Personal data comes out of the free text before it goes to an external API, the prompt and model version are recorded, and a sample of answers coded by a person checks that the categories still hold.
• The comparison agreed before launch: the baseline window, the change being measured and the list of other releases in the same period. If the traffic allows it, a holdout group as well.

Try this on your own funnel

Pick your highest-value funnel and list every page a customer can see between the basket and the confirmation page. Then check that each one sends an event. If the two lists do not match, that is the first finding to take to the redesign meeting.

Travel funnels from the other side: buses in Santiago and a flight to Rio

Part of Travel funnels from the other side, a series on what I learn as a traveller about the booking funnels I usually measure.

In November 2024 I travelled through Chile, from Santiago down to the northern part of Chilean Patagonia, and then flew from Santiago to Rio de Janeiro. I have spent years measuring checkout funnels. On this trip I was the customer who dropped out of them, more than once.

The bus that kept disappearing

The hardest part of the trip was buying bus tickets in Santiago. On the platform I was using, buses appeared in the results and then disappeared. When I found one, the form asked for a RUT, the Chilean tax ID that a visitor does not have, so the purchase never finished.

Later I learned what I had been using. Recorrido and Kupos are marketplaces that resell tickets from many bus companies, and Recorrido has belonged to the Canadian company Busbud since 2022 (LatamList). Neither is a bus company's own site or an official government service, and each works with different operators, which is why they showed me different types of bus. Other travellers describe the same RUT wall (Tour du Mondiste).

In the end, everyone I asked gave me the same advice: go to the Pajaritos terminal and buy the ticket there, and avoid Estación Central. The advice has a basis. Carabineros list the Estación Central metro station and the Santiago bus terminal among the city's theft hotspots (Meganoticias, 2022). Pajaritos is a smaller terminal in Lo Prado, on the west side of the city.

The flight I booked through someone else

For the flight from Santiago to Rio, the airlines' own websites were my main problem. Again I could not complete all the data they asked for. I ended up booking through Booking.

What the companies' dashboards saw

From the inside, none of this looks like a story. The bus companies saw sessions that searched and left. The resellers saw a form abandoned at the data step. The airline saw an abandoned checkout and, somewhere else in its reports, a sale through an online travel agency. Nobody saw a foreign customer who wanted to buy and could not.

It reminded me of a checkout I worked on at JD Sports Group, where the numbers looked wrong because one page in the funnel was never tracked. Here the hole was bigger: part of the journey happened on sites the companies did not control, and part of it happened at a ticket window.

What I would measure

If I were the analyst at one of these companies, I would start with four things.

1. Form errors by field, split by the visitor's country. A required field that foreigners cannot fill in shows up as one field with a much higher error rate for one group of visitors.
2. The stability of inventory across searches and channels. The same route and date, searched twice within a few minutes on the company's own site or on a marketplace, should return the same buses. When a departure disappears from one channel and stays on another, that is an inventory problem, and customers read it as unreliability.
3. Demand that leaks to other channels. Compare abandoned checkouts from a market with the bookings from that market that arrive through intermediaries or at the ticket office. If one goes up when the other goes up, the direct channel is losing sales it could keep.
4. One short question on exit: what stopped you? At JD a short survey on the checkout gave us reasons that the funnel numbers alone could not.

A question to finish

When a customer gives up on your checkout and buys somewhere else, does any report connect the two events, or do they show up as one lost session and one unrelated sale?