How to evaluate a hospitality retention pilot with a comparison group

A new buddy programme launches in three hotels. Early retention improves. Before extending it across the group, there is a harder question: how much of that improvement would have happened anyway?

A retention pilot evaluation needs a credible comparison, consistent measurement and evidence that the intervention actually reached employees. Build those elements before launch, while choices about timing, sites and data can still be changed.

Write the decision before choosing the metric

Specify what leaders must decide: stop, adapt, extend the test or roll out. Then state the proposed mechanism in plain language. For example, trained buddies may help new starters resolve practical problems earlier, reducing avoidable departures during their first 90 days.

Choose one primary outcome, such as continuous group employment through day 90 among eligible starters. Fix the organisation boundary, seasonal-contract rules, intake window and reporting cut-off. List secondary measures separately: delivery quality, new-starter experience and operational burden.

Agree what would count as a practically worthwhile improvement before seeing the results. The threshold should reflect the organisation’s circumstances and the uncertainty it can tolerate, not a universal percentage taken from another employer.

Choose a comparison that could plausibly answer the question

Where feasible, allocate eligible properties or teams randomly to the pilot and comparison conditions. Shared managers and shared rosters can make individual allocation impractical, so the property may be the more sensible unit. An analyst should help determine how many units and starters are needed; many employees in only two hotels do not create many independent hotel-level comparisons.

HM Treasury’s guidance on quality in impact evaluation explains why random allocation and strong comparison designs support more credible attribution than a simple before-and-after result. Randomisation reduces systematic selection, but a small trial can still produce imbalanced groups by chance.

If random allocation is not possible, select comparison hotels before reviewing post-pilot outcomes. Examine similarities in:

• Previous retention levels and trends across several comparable intakes.
• Location, local recruitment conditions and seasonality.
• Role mix, contract mix and proportion of first-time hospitality recruits.
• Opening or refurbishment status, management changes and occupancy pressure.
• Pay, scheduling practices and other programmes affecting new starters.

A city business hotel may be a poor comparator for a seasonal resort even if their headcounts match. Record important differences rather than calling the sites “matched” on the strength of one similarity.

Use the same observation window

Follow pilot and comparison starters for the same duration. For a 90-day outcome, the evaluation cannot finish until the final included starter has had the opportunity to reach day 90. Someone still employed at day 25 is not yet a 90-day retention success.

Keep early leavers in the eligible cohort. Do not analyse only employees who completed buddy sessions, because completion may itself depend on staying. If the design allocates employees or sites to an intervention, preserve an analysis by original allocation, while reporting actual participation separately.

Decide how transfers, rehires and missing employment records will be handled in advance. Apply those rules to both groups. Differences in record-keeping can otherwise masquerade as differences in retention.

A worked difference-in-differences example

Hypothetical example, not a research finding. Each of the following four cells contains 100 eligible starters with complete 90-day follow-up:

• Pilot properties before launch: 70 retained, giving 70% retention.
• Pilot properties after launch: 82 retained, giving 82% retention.
• Comparison properties before launch: 75 retained, giving 75% retention.
• Comparison properties after launch: 79 retained, giving 79% retention.

The pilot group’s improvement is 12 percentage points. The comparison group’s improvement is four percentage points. The difference in those changes is 12 − 4 = eight percentage points.

Under the design’s assumptions, that is the estimated additional change associated with the pilot. For 100 post-launch starters, eight percentage points corresponds to eight additional retained starters relative to the estimated counterfactual. It is not an 8% relative improvement, nor proof that eight identifiable individuals stayed because of the programme.

The key assumption is that, without the pilot, outcomes in the two groups would have changed similarly. The Magenta Book’s analytical methods annex discusses this difference-in-differences approach and the use of pre-intervention trends. Similar past trends support the argument but cannot prove what would have happened afterwards.

Investigate rival explanations

Keep a contemporaneous change log. A pay rise at pilot properties, a new competitor recruiting near comparison sites or a change in contract mix could affect the result. Record the timing and which employees were exposed.

Look for spillovers. If comparison managers adopt the same buddy practices, the study is no longer comparing the planned conditions. If experienced colleagues are moved into pilot hotels to support delivery, resulting staffing changes elsewhere may also matter.

Check for regression to the mean. Sites chosen because they had one unusually poor intake may improve even without intervention. Several pre-pilot cohorts are more informative than a single low starting point.

Report uncertainty rather than declaring success from the point estimate alone. Ask an analyst for an interval appropriate to the design, including clustering where allocation occurs by property. A wide interval may leave both worthwhile benefit and little effect plausible. “Not statistically significant” does not establish that the programme has no effect.

Check delivery, cost and unintended consequences

Use a short implementation record: eligible starters, buddy assigned, first meeting completed, later contact completed, support hours and exceptions. Sample the quality of contact; an automated tick does not establish that someone received useful help.

Keep programme spending and staff time visible without turning speculative avoided exits into guaranteed savings. Watch for workload displaced onto buddies, unequal access for night teams or reduced support for established staff. Normal protections and essential support should not be withheld to create a comparison.

Related WINC Wire reading on onboarding and culture and rota-related retention pressures can help frame operational questions, but neither substitutes for evidence from your own test.

Make the conclusion match the evidence

Write the decision paper around five items: what was delivered, what changed, the estimated difference against comparison, what could still explain it and the next decision. State counts, definitions and uncertainty alongside the headline.

Roll out when the evidence and delivery case justify it. Adapt when the mechanism looks promising but execution was weak. Extend a test when meaningful uncertainty remains and further observation can resolve it. A disciplined pilot is valuable even when it prevents an attractive but unsupported programme from becoming permanent.

Sources

• HM Treasury: Quality in policy impact evaluation
• HM Treasury: Magenta Book Annex A analytical methods

AI-generated editorial illustration.