Agree the metric before you build the dashboard

At JD Sports Group, marketing and ecommerce each had their own figures and neither trusted the other's. The programme that fixed it started with definitions agreed by the managers, and the dashboards came afterwards.

The situation

From 2021 to 2024 I was Senior Digital & Marketing Data Analyst at JD Sports Group and product owner of a programme of 8 projects to improve measurement end to end. When the programme started, each manager had their own objectives and their own KPIs. Marketing did not trust the ecommerce figures, ecommerce did not trust the marketing ones, and meetings that should have ended in a decision ended in a discussion about whose report was right.

The request that usually arrives in that situation is a new dashboard. Built on top of the disagreement, it would have shown both sets of numbers on one screen, and the discussion would have carried on in front of it.

What I did

I started with a measurement plan, before choosing any tool. I ran sessions with the managers to agree the business objectives first, and then the KPIs that would show whether we were meeting them, for internal users and for customers.

To give it a structure I used the GDS Performance Framework, which separates primary and secondary metrics. Each area ended up with a small number of metrics that mattered, each with one shared definition behind it. GOV.UK has since withdrawn that framework page and points readers to its Service Manual guide on measuring success, which asks teams to work out their measurements while they build the service.

I split the roadmap in two. The strategic part was objectives, metric definitions and ownership. The operational part was instrumenting the tracking with developers and building dashboards. We ran it as an agile programme, with a Trello board for the programme and Jira for the delivery work with developers. New requests arrived every week, so the backlog kept moving, and I prioritised by dependency and by how many teams a piece of work would unblock. A metric was defined before it was implemented.

Dashboards were released one implementation at a time. After each one, people could see an operational dashboard, use it and tell us what was missing, and most of the refinement of the metrics happened at that stage. When a new implementation moved into business as usual, a short survey checked whether it met the need it had been built for.

The programme delivered its 8 projects with 90% stakeholder satisfaction, and adoption of analytics grew 70% year on year. Adoption is the figure I value most, because it shows that people started coming to the data to settle questions.

What I learned

Implementing a metric that nobody has agreed on moves the argument downstream. In a meeting the disagreement is visible and cheap to resolve. Inside a dashboard it looks like a data quality problem, and tracing it back to a definition takes much longer.

The second lesson came from releasing early. A definition agreed in a workshop is still a draft until people use it with real data, and the first dashboard was the cheapest place to find out what the draft had missed.

What I would do today

Creating a metric keeps getting easier. With Tag Manager templates and point-and-click tagging, a new event can be live in minutes. When an event takes minutes to create, the definition is the part of the work that needs an owner.

If I ran the same programme now, I would add four habits.

1. A written definition for every primary metric, with the business question it answers, the formula, the data source, the filters, the owner and the date it was agreed.
2. A change log that the dashboard shows. When a definition changes, the date appears on the chart, so nobody reads a step in the line as a change in the business.
3. No new event in the tracking plan unless it feeds a metric that has a definition.
4. A review of the secondary metrics every quarter, to retire the ones nobody opens.

Before your next dashboard

Take the headline metric of the dashboard you are about to build and ask two managers who use it to write down, separately, how it is calculated. If the answers differ, that conversation comes first, and the dashboard can wait a week.

A hole in the checkout funnel: fix the tracking before you redesign the page

At JD Sports Group the business was ready to redesign the checkout because the funnel looked broken. The first problem I found was a page nobody had tracked, and fixing it changed what the rest of the project could do.

The situation

I was Senior Digital & Marketing Data Analyst at JD Sports Group from 2021 to 2024. One project started with a clear request: the checkout funnel is broken and needs redesigning. Several teams had a view on what was wrong and what to change, and the pressure was to start changing things.

What I did

I started with the analysis. Compared with benchmark, the funnel numbers were off in a way that did not fit a checkout that had not changed recently. Following the drop led me to a page that had been added to the checkout at some point and never tracked. Customers were passing through a step the funnel could not see, which left a hole in the report.

We fixed the tracking first. That gave the teams a funnel they could trust, and it gave me the credibility to ask for more advanced tracking, which was approved.

The numbers then showed where people left the checkout, but they could not say why. I added a survey with Hotjar, the tool the team already had, and used the OpenAI API to clean and categorise the open-text answers. Payment problems came out as the main theme, backed by what customers had written.

Session recordings came next, but the volume was too high to turn into a priority list. So I changed approach and worked with Customer Care to put live chat on the checkout during working hours, on one condition: agents logged the reason for each conversation in Salesforce. That gave us the reasons directly from customers, every week.

There were competing requests. Marketing wanted Customer Care to run video calls with customers, and Customer Care wanted the chat on product pages. I argued for the checkout first by walking both teams through the funnel step by step, showing what data we already had at each stage and what each option would tell us that we did not know yet. The reporting stayed simple, with charts and a weekly report, and the teams started reusing it in their own meetings. Once the report was useful to them, the priority was no longer up for debate.

The result, and how it was measured

Checkout conversion improved by 4% (relative) after the changes. That figure comes from a before-and-after comparison against the baseline before launch. There was no A/B test, so the comparison cannot separate the effect of the changes from seasonality or from anything else that moved in the same period. When I present the number, that limit goes next to it.

The weekly report outlived the project, and Customer Care was still using it in their own meetings long after it closed. It also showed me the weak point. As Customer Care hired new people, the logging discipline dropped and the reason data became less reliable, so the reports kept coming out while their value went down. Retraining became a recurring task, built into how the report was maintained.

What I learned

A funnel is a data product, and the cheapest data quality control is checking that every step in it is measured before anyone draws conclusions from it. The epilogue taught me the second lesson: data that depends on people typing a reason into a CRM needs the same upkeep as a pipeline, because adoption work does not end at handover.

What I would do today

I would add four things.

• Tracking in the definition of done. A new checkout step does not go live until its event is in the tracking plan and a test confirms that the event arrives.
• An automated completeness check on the funnel. Pages inside the checkout path that receive sessions and send no funnel event should raise an alert.
• The LLM categorisation treated as a measurement instrument. Personal data comes out of the free text before it goes to an external API, the prompt and model version are recorded, and a sample of answers coded by a person checks that the categories still hold.
• The comparison agreed before launch: the baseline window, the change being measured and the list of other releases in the same period. If the traffic allows it, a holdout group as well.

Try this on your own funnel

Pick your highest-value funnel and list every page a customer can see between the basket and the confirmation page. Then check that each one sends an event. If the two lists do not match, that is the first finding to take to the redesign meeting.

Travel funnels from the other side: buses in Santiago and a flight to Rio

Part of Travel funnels from the other side, a series on what I learn as a traveller about the booking funnels I usually measure.

In November 2024 I travelled through Chile, from Santiago down to the northern part of Chilean Patagonia, and then flew from Santiago to Rio de Janeiro. I have spent years measuring checkout funnels. On this trip I was the customer who dropped out of them, more than once.

The bus that kept disappearing

The hardest part of the trip was buying bus tickets in Santiago. On the platform I was using, buses appeared in the results and then disappeared. When I found one, the form asked for a RUT, the Chilean tax ID that a visitor does not have, so the purchase never finished.

Later I learned what I had been using. Recorrido and Kupos are marketplaces that resell tickets from many bus companies, and Recorrido has belonged to the Canadian company Busbud since 2022 (LatamList). Neither is a bus company's own site or an official government service, and each works with different operators, which is why they showed me different types of bus. Other travellers describe the same RUT wall (Tour du Mondiste).

In the end, everyone I asked gave me the same advice: go to the Pajaritos terminal and buy the ticket there, and avoid Estación Central. The advice has a basis. Carabineros list the Estación Central metro station and the Santiago bus terminal among the city's theft hotspots (Meganoticias, 2022). Pajaritos is a smaller terminal in Lo Prado, on the west side of the city.

The flight I booked through someone else

For the flight from Santiago to Rio, the airlines' own websites were my main problem. Again I could not complete all the data they asked for. I ended up booking through Booking.

What the companies' dashboards saw

From the inside, none of this looks like a story. The bus companies saw sessions that searched and left. The resellers saw a form abandoned at the data step. The airline saw an abandoned checkout and, somewhere else in its reports, a sale through an online travel agency. Nobody saw a foreign customer who wanted to buy and could not.

It reminded me of a checkout I worked on at JD Sports Group, where the numbers looked wrong because one page in the funnel was never tracked. Here the hole was bigger: part of the journey happened on sites the companies did not control, and part of it happened at a ticket window.

What I would measure

If I were the analyst at one of these companies, I would start with four things.

1. Form errors by field, split by the visitor's country. A required field that foreigners cannot fill in shows up as one field with a much higher error rate for one group of visitors.
2. The stability of inventory across searches and channels. The same route and date, searched twice within a few minutes on the company's own site or on a marketplace, should return the same buses. When a departure disappears from one channel and stays on another, that is an inventory problem, and customers read it as unreliability.
3. Demand that leaks to other channels. Compare abandoned checkouts from a market with the bookings from that market that arrive through intermediaries or at the ticket office. If one goes up when the other goes up, the direct channel is losing sales it could keep.
4. One short question on exit: what stopped you? At JD a short survey on the checkout gave us reasons that the funnel numbers alone could not.

A question to finish

When a customer gives up on your checkout and buys somewhere else, does any report connect the two events, or do they show up as one lost session and one unrelated sale?

Product Analytics: How to calculate retention and what does it means.

Understanding how well your business keeps customers engaged is crucial for success. Retention rate is a key metric to measure this success, but there's no one-size-fits-all approach. Depending of your goals and therefore your KPIs you can choose one of them to measure strategies or tactics.
Let's explore three common retention methodologies:

Classic Retention (Day N)

This method tracks the percentage of new users who return on a specific day. For example, if 3 out of 10 new users return on the second day, your Day 2 retention rate is 30%.

• Benefits: Daily granularity, easy to explain and calculate.
• Limitations: Sensitive to daily fluctuations.
• Best for: Measuring the immediate impact of short-term campaigns or analyzing day-to-day retention trends.

Range Retention

This measures retention over a specific time interval, like a week or a month. For instance, if 9 out of 50 new users return within the first week, your first-week retention rate is 18%.

• Benefits: Smooths out daily noise, easy to explain, good for long-term trend analysis.
• Limitations: Less granular, longer lag time for results.
• Best for: Monitoring overall business health and identifying weekly or monthly behavioral patterns.

Rolling Retention

This calculates the percentage of new users who return on or after a specific day. It provides a single metric representing long-term customer engagement.

• Benefits: Fast to calculate, reflects overall stickiness.
• Limitations: Open-ended, can be constantly changing, doesn't differentiate between frequent and infrequent users.
• Best for: Supplementing other KPIs and understanding long-term customer relationships.

Choosing the Right Method

• Classic Retention: Ideal for tracking short-term campaigns and daily engagement.
• Range Retention: Best for monitoring overall business health and identifying behavioral patterns.
• Rolling Retention: Useful for supplementing other metrics and understanding long-term customer relationships.