At Spicy Mango, we continuously strive to enhance our products to meet the evolving needs of our clients. Our sports data platform, Gameday, has always excelled at ingesting, transforming, and distributing sports data to businesses via REST APIs and webhooks. However, as the demand for real-time sports updates for direct-to-consumer (D2C) applications grew, we recognised the need for a low-latency, scalable solution. This led us to explore WebSocket-based pub/sub mechanisms to efficiently distribute live updates, such as goals scored or racing car/bike position, to millions of connected clients.

In this article, we’ll talk you through:

• Evaluation of Websocket technologies

• The work we did to onboard and trial

• The rationale for opting to shift to our own Websocket implementation

• The associated challenges and benefits it’s brought

The Need for Dedicated WebSocket-Based D2C Capability

Sports fans expect instant updates. Latency is no longer measured in seconds but in milliseconds. Our existing distribution methods - REST APIs and webhooks - are excellent for structured data exchange but are not designed for real-time push updates at scale. WebSockets, on the other hand, provide a persistent, bidirectional communication channel, making them ideal for delivering instantaneous updates to millions of clients worldwide.

To support this transition, we evaluated multiple approaches, including off-the-shelf protocols like MQTT, before ultimately developing our own WebSocket-based pub/sub solution in Node.js.

Trialling MQTT: A Powerful but Unsuited Solution

MQTT is a well-established publish-subscribe protocol, widely used in IoT applications and real-time messaging. On paper, it seemed like a strong candidate for our needs. It supports low-bandwidth environments, has built-in quality-of-service (QoS) levels, and facilitates efficient message routing. However, during our trial, we encountered several challenges that made it challenging for Gameday’s D2C use case.

1. Complex Configuration and Management at Scale

   1. Deploying MQTT at scale proved to be cumbersome. Managing multiple broker instances, ensuring high availability, and scaling up dynamically in response to traffic spikes required significant operational effort. Our DevOps team had to invest time in fine-tuning configurations, handling cluster management, and optimising load balancing, which added unnecessary overhead.

   2. We also investigated third-party managed solutions - such as HiveMQ - instead of managing our own Mosquitto instances. However, the cost at scale, quickly made it an unviable option for our needs.

      
      

2. Difficult Monitoring and Reporting

   1. While MQTT supports message tracking and logging, it relies heavily on MQTT messages themselves for monitoring. MQTT brokers often utilise special topics, such as the $SYS topic, to publish internal status and metrics. Monitoring these brokers involves subscribing to these topics to gather information about the broker's health and activity. This approach means that monitoring tools must act as MQTT clients, subscribing to these internal topics to collect metrics, which can add complexity to the monitoring setup. While this method is effective within the MQTT ecosystem, it may not integrate seamlessly with external monitoring systems that expect metrics via standard protocols like HTTP. This made real-time reporting, debugging, and analytics more challenging than anticipated. We required a system that provided deeper observability into connection health, delivery success rates, and latency metrics - features that were either complex to implement or not natively available in MQTT.

      
      

3. Lack of Flexible Authentication and Authorisation

   1. One of the most critical drawbacks was the inflexibility in managing authentication and authorisation. MQTT brokers offer authentication mechanisms, but implementing fine-grained access control for different classes of users - such as differentiating between premium and free-tier subscribers - proved complex. Additionally, our reliance on third-party plugins within Mosquitto made us uncomfortable, as it introduced an external dependency that we had limited control over. Furthermore, we observed a decline in performance as our complex authentication and authorisation requirements necessitated offloading and proxying these processes to a separate Spicy Mango-developed and maintained service. This added unnecessary latency and complexity, reinforcing our decision to pursue a more integrated and efficient solution.

      
      

4. Not Aligned with Our Development Stack

   1. While MQTT is robust, its ecosystem is primarily geared towards IoT and industrial applications, often with brokers written in languages like C (Mosquitto), Erlang (emqttd), or Java (Moquette). Our engineering team primarily works with Node.js for backend development, and maintaining an MQTT solution required either introducing new technology into our stack or relying on third-party brokers, neither of which was ideal for long-term maintainability.

      
      

5. Unnecessary Overhead for Our Use Case

   1. MQTT is designed for a broad set of real-time messaging use cases, including those with constrained devices and unreliable networks. While it excels in those scenarios, our primary goal was efficient, scalable real-time distribution of sports metadata over stable internet connections. The additional features of MQTT, such as message retainment and complex QoS handling, added unnecessary overhead and complexity to our infrastructure without providing proportional benefits.

Developing Our Own WebSocket-Based Pub/Sub Solution

After thoroughly evaluating MQTT, we made the strategic decision to build our own WebSocket-based pub/sub system using Node.js. This decision allowed us to optimise for our specific needs while keeping our technology stack consistent and maintainable. Here’s why our in-house solution proved to be the right choice:

1. Seamless Integration with Gameday’s Existing Architecture

   1. Our platform already runs on Node.js, making it straightforward to develop a WebSocket service that integrates seamlessly with our existing microservices. This reduced onboarding time for our engineers and simplified ongoing maintenance.

      
      

2. Lightweight and Optimised for Sports Metadata

   1. By controlling the implementation, we could optimise message structures and payload sizes for sports data. Unlike MQTT, which supports various QoS levels that we did not require, our WebSocket solution delivers updates with minimal processing overhead.

      
      

3. Scalable and Easier to Manage

   1. With our own solution, we could leverage AWS services such as Elastic Load Balancing and auto-scaling features to ensure smooth scaling. We built monitoring tools tailored to our needs, generated real-time insights into connection stability, active subscribers, and message delivery times, and also integrate to our standard logging framework.

      
      

4. Custom Authentication and Authorisation

   1. Security was a primary concern, and our bespoke solution enabled us to implement fine-grained authentication and authorisation controls. Our authentication and authorisation system is powered by AuthIDConnect, a Spicy Mango-developed solution. AuthIDConnect supports registered users logging in via passwords or TOTP, issuing short-lived access tokens and long-lived refresh tokens (both signed JWTs). Users can refresh their access tokens using the refresh token, maintaining secure and continuous access. The access token contains claims defining the user’s authorisations.

   2. AuthIDConnect also supports anonymous users, a crucial feature for third-party businesses integrating Gameday sports metadata into their own solutions. In this model, when an anonymous end-user is authorised to use the third-party service, the authentication request is proxied by the third party to AuthIDConnect with a session ID. AuthIDConnect then issues a signed anonymous access token containing the session ID, which the end-user uses to authenticate and authorise their access to Gameday’s APIs and WebSocket pub/sub service.

   3. Our WebSocket solution is designed to validate these tokens in an efficient and speedy manner, ensuring minimal latency and a seamless experience for all connected clients.

      
      

5. Reduced Infrastructure Complexity

   1. By avoiding an additional protocol layer, we eliminated unnecessary dependencies and simplified operational overhead. Our WebSocket service sits within our existing AWS infrastructure, reducing the need for additional maintenance and third-party dependencies.

      
      

6. Meeting the Pillars of a Well-Architected Framework

   1. Our solution aligns with the key pillars of AWS's Well-Architected Framework:

      1. Operational Excellence: Simplified management and monitoring with custom-built observability tools.

      2. Security: Fine-grained authentication and authorisation to control access and prevent unauthorised data exposure.

      3. Reliability: AWS auto-scaling and managed WebSocket APIs ensure high availability and resilience.

      4. Performance Efficiency: Minimal processing overhead and optimised message structures improve latency and response times.

      5. Cost Optimisation: Eliminating third-party dependencies reduces licensing costs and infrastructure overhead.

      6. Sustainability: Reduced operational complexity and efficient resource usage help maintain a sustainable architecture.

A 10,000ft View

A (very) high level picture of what we created is below. Crucially, we can manage a scale the web socket distribution solution across many Availability Zones and Regions, whilst authentication and authorisation is all carried in the AuthIDConnect token. If we want, we can deploy in a fan-out pattern of origin and edge servers, to scale to meet demand in an any region - all with the same WebSocket server implementation.

Lessons Learned and Final Thoughts

Our journey to a WebSocket-based D2C solution taught us valuable lessons. MQTT is a powerful and versatile protocol, well-suited to IoT and machine-to-machine communication. However, for Gameday’s specific requirements—scalability, low-latency messaging, integration with our Node.js stack, and fine-grained security controls—our custom-built WebSocket solution was the best fit.

This experience reinforced the importance of selecting technologies that align with our core expertise and operational goals. By developing our own solution, we now have a highly optimised, scalable, and secure system that delivers real-time sports metadata updates to millions of users with minimal latency.

As Gameday continues to evolve, we remain committed to exploring new ways to enhance our platform. Whether it’s further optimising our WebSocket infrastructure, incorporating edge computing, or leveraging new cloud technologies, our goal remains the same: to provide the best real-time sports data experience possible.