How to Tell Whether Your Loyalty Program Is Working

Almost every loyalty app shows a chart where members spend far more than non-members. It looks like proof. It usually isn't.

Your best customers are the ones most likely to join a loyalty program. They were going to spend more anyway. Comparing members with non-members mostly measures who joins, not what the program changed. To know whether a program earns its cost, you need a fairer comparison.

The Problem With "Members Spend More"

Imagine a store where regulars order every month and one-time buyers never return. The regulars all join the program; the one-time buyers mostly don't. Members will spend many times more than non-members even if the program did nothing at all. Any report built on that comparison will credit the program with sales it didn't cause.

The same trap catches email results. A win-back email sent to customers who were about to reorder anyway will look like it brought them back.

Use a Held-Back Group

The fix is the same one medicine uses: a control group. Pick a small random share of members, often about one in ten, and leave them out of the features you want to measure. They keep the basics, such as earning and redeeming points, but don't get the extras: the win-back emails, bonus campaigns, upsells or games.

Because the group is random, it looks like everyone else in every way that matters. Any difference in what the two groups spend afterwards comes from the features, not from who they are. That difference is the program's real effect, often called lift.

Three rules keep the comparison fair:

• Assign the group by a fixed rule, such as a hash of the customer's ID, so a customer is always in the same group.
• Don't change the share while you measure. Moving people between groups mixes the results.
• Check the split before you trust it. Before any new feature is live, both groups should spend about the same. If they don't, something in the setup is off.

Give It Time

A held-back group on a small store is small, so its numbers move around from week to week. A store with a few hundred orders a month may need several months before the difference is clear. Read the trend over that period rather than a single week, and be wary of any result that swings from big to nothing between reports.

Estimated, Attributed and Confirmed Revenue

Loyalty reports mix three kinds of numbers. Knowing which is which keeps you from overstating results:

• Confirmed revenue is what actually happened: orders placed and paid, minus refunds.
• Attributed revenue is confirmed revenue linked to something the program did, such as an order that used a reward code or an upsell added from a recommendation. The order is real; the credit is a rule.
• Estimated revenue is a projection, such as the extra sales the lift suggests over a year. It is useful for planning, but it is not money in the bank, and it is not a guarantee.

A good report labels each number clearly and keeps estimates conservative. If a figure depends on an assumption, the report should say so.

What to Look At Each Month

1. Lift: extra spend per member in the treated group over the held-back group.
2. Repeat rate: the share of first-time buyers who place a second order, in each group.
3. Redemptions: how many rewards were used, and the discount they cost you.
4. Program cost against lift: the app fee plus discounts given, compared with the extra margin the lift brings in.

If the extra margin covers the cost, the program pays for itself. If it doesn't after a fair period, change the rewards or the timing and measure again.

LumiLink Loyalty holds back a random tenth of members on Growth and compares the two groups in its results report, with estimates labelled as estimates.