Virtual Referee for eSports Streaming

An eSports streaming company had a healthy and growing base of players who would stream matches publicly. Streaming boosted engagement by giving players something to watch when they weren't playing, engaged new audiences, and provided transparency. High value matches were scheduled and refereed by human officials, but lower value and impromptu matches were too numerous to staff for. Extrasensory¹ was engaged to onboard an AI-based "virtual referee" that could follow the progress of a match in real time and intervene in case of irregularities, greatly extending the capabilities of the human officials.

Motivation

Most matches were played fairly and sincerely, but glitches or player errors led to a steady stream of cases needing intervention. The overwhelming majority of the time, players needed relatively simple instruction or correction to complete their match. As a result, the primary time-consuming activity for officials was watching and checking in on matches. Human officiation was thus boring and repetitive, sapping energy needed for important judgement calls.

Approach

Extrasensory began "in the film room", collecting examples of the types of interventions needed and the circumstances that led up to them. Based on these examples, we discussed with officials the "contract" that virtual referees should have with their human counterparts: When to intervene, raise an alert, or report errors. Based on an understanding of this handoff, Extrasensory developed a solution as follows:

• Selection of an LLM with vision capabilities so that it could be given natural language instructions and taught about video frames.
• Fine-tuning of the base LLM model using labeled video frames to teach it the specific subject matter.
• Development of a cloud-based controller to keep track of matches and live streams, assign virtual referees to them, and route alerts as appropriate.

Results

With the solution developed, deployed, and tested, the virtual referee was ready for action. Over the course of a one month testing period it exceeded expectations for accuracy, response time, trainability, and scope. Automated officiation was on a path to incremental adoption, and management was given confidence to take the next step.