Competitive Matchmaking in Online Gaming

A gaming app was looking for ways to enhance the quality of competition among their players. Extrasensory¹ brought mathematical modeling and statistical analysis to product, design, and engineering functions to develop a skill rating feature from concept to release, giving management confidence that their concerns were addressed at each step of the way.

Motivation

Players in the network represented a range of skill levels which had widened as the player base grew, but players had poor indicators of the skill when choosing opponents. To support further growth, the network needed to assist players in finding match-ups that would be fair or competitive. This was a particular problem for new players who were being taken advantage of by "sharks", limiting growth.

Approach

Extrasensory began by studying the interaction of business strategy and existing and prospective player personas. With this overview in mind, a historical analysis of the network's player behavior led us to suggest a feature that would track and communicate player skill. This feature was developed via iteration over the following:

• Development of a skill rating algorithm similar to Elo ratings used in chess and other online games.
• Backtesting and statistical validation of algorithm performance against experiences for key personas.
• Alignment of the presentation layer for the algorithm with product and design to ensure meaningful communication of skill ratings.
• Presentation of proposed changes to engineering including seed data, pseudo-code, test cases to preemptively resolve all ambiguities and minimize guesswork in implementation.
• Monitoring of user behavior during and following release to verify that intended effects and alert on any unintended consequences with special attention paid to the new user experience.

Results

With Extrasensory's guidance, the feature was implemented quickly and released as designed. Immediate adoption by users indicated that assumptions about identified personas were valid, and evaluation at 30 and 60 day milestones confirmed improvements projected in backtesting.

Players were accurately and straightforwardly informed about the skill of prospective opponents, allowing a much wider range of skill levels to compete. These additional "divisions" naturally contained more players, greatly expanding the addressable market.

New players were given instructions to help them identify and avoid "sharks", and sharking was effectively eliminated. This resulted in improved retention for new players.

As a bonus, the skill rating model offered a measure of statistical anomaly which was developed into an internal indicator of fraud. Deviations from the statistical signature of natural, competitive play could be detected and alerted on automatically.