Migrating from Kafka to RabbitMQ at Simplebet

Why and How

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Who am I - Dave Lucia

  • VP, Engineering at Simplebet ๐Ÿ€ โšพ ๐Ÿˆ
  • โค๏ธ Elixir
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

๐Ÿถ ๐Ÿถ dad

๐Ÿ‘ถ dad

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187


Our mission is to power the future of fan engagement

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Make every moment a of a sporting event a betting opportunity

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Micro Markets ๐Ÿˆ โšพ ๐Ÿ€

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Odds as a service

Powered by Machine Learning ๐Ÿค–

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Craft branded end-user experiences

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Data Pipeline

  1. ๐Ÿ“ก Receive incident data
  2. ๐Ÿ“ˆ Create markets
  3. ๐Ÿ”ข Produce odds ๐Ÿค–
  4. ๐Ÿš€ Publish markets
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

In the beginning...

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

The Problem

w:600px

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Serving odds

w:400px

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Market Publishing Requirements

  • โœ… Customizeable per customer
  • ๐Ÿ‘จโ€๐Ÿ‘งโ€๐Ÿ‘ง Distributeable
  • ๐ŸŽ๏ธ Minimal latency
  • ๐Ÿ› ๏ธ Durable
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Kafka

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

left w:550px

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

w:500px

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

โš ๏ธ Dilemmas โš ๏ธ

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Moving Parts

  1. Kafka Broker
  2. Kafka Rest Proxy
  3. Schema registry
  4. Kafka UI
  5. Zookeeper
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Partioning / Topic Strategy

  • Topic per customer?
  • Topic per match?
  • Topic per league?
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Delivering data a la carte

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Data Retention

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Authentication and Authorization

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Customer Complexity

  • Avro / Schema Registry
  • Managing offsets
  • Separation of integration data
  • Kafka Protocol
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Summary

  • Client needs to know Kafka
  • Multiple points of failure
  • Time to market
  • Adding complexity
  • Hard to use
  • Expensive
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Re-evaluating our Integration

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187
  • Integrations should be familiar
  • Customizeable
  • Easy to provision
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187
  • AMQP
  • RabbitMQ Management Interface
  • Plugin-based ecosystem
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

w:400px

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Serialization

  • JSON
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Authentication and Authorization

  • RabbitMQ Management Interface
  • V-Host per consumer
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Data Retention

  • Durable / Quorum queues
  • Reads are destructive
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Data a la carte

  • Publisher produce to a fanout exchange
  • E2E bindings
  • Headers dictate the flow of data
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Topic / Partioning Strategy

  • Let the customer decide!
  • Headers exchange bindings
  • Headers dictate the flow of data
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

AMQP

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Summary

  • Batteries included
  • Lower cost
  • Higher reliability
  • Easier integration
  • Gentle learning curve
Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

โค๏ธ

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

We're hiring

https://jobs.lever.co/simplebet

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

Questions?

Kafka ๐Ÿ‘‰ RabbitMQ at Simplebet โ€” @davydog187

My Notes

Explain who our customers our (Enterprise) Sport Coverage Real-time nature Describe how we use machine learning 1. Years of historical data 2. Build features based of deep understanding of the game 3. Refer to the ML panel

Explain what our free to play product is

The last step, publish markets, we were not focusing on

* Story starts in the early days * We were building our initial archicture of the odds feed * Many open questions

* Needed a means to publish our data * Focused on building the core element of our product * Integration with Machine learning * Building a Rust ML framework

TODO a picture of our feed at a high level TODO highlight filtering an distribution

Apache Kafka is an distributed event streaming platform for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

These are the values of Kafka that attracted us to Kafka initially * Reputation for being fast * Nascent customer integration story Highlight that we picked Kafka because it seemed like a good enough fit, but we really did not understand the needs of our customers yet

You can use a managed solution BUT! you still need to deal with complexity locally Deploying per customer * Deployed a broker per customer * Terraformed out a topic per customer

* Customer packages (nfl \ mlb \ different markets)

* Once a message is written, it is readable for a long time after * This means that stale messages remain until they are dropped from the log by the broker

RBAC control is offered for Kafka via the Confluent Metadata service

* GraphQL * Websockets/streaming http still need to build intermediate

Running in production since June of last year We now have multiple customers on rabbit Easy to provision new ones We've expanded the use of Rabbit internally