Behind the curtains

I recall having read Martin Kleppmann's impressive book on Data Intensive Applications, which included a noteworthy observation about Twitter's engineering team: Twitter had created different back-end systems for different sets of accounts. Specifically, Twitter employed isolated databases and back-end services for highly followed accounts (e.g. Elon Musk, Justin Bieber, and Kim Kardashian.) This approach ensured that whenever these accounts tweeted, their updates would appear in users' timelines quickly.

And this made a lot of sense at the time; for sure you would not handle your neighbors tweet the same way as the barrage of tweets from a celebrity. Here's a quote from the book:

Most users’ tweets continue to be fanned out to home timelines at the time when they are posted, but a small number of users withaverylargenumber of followers are excepted from this fan-out. Instead, when the home timelineisread,the tweets from celebrities followed by the user are fetched separately and mergedwiththehome timeline when the timeline is read

We decided to choose a similar approach for our own system. We created a separate set of databases and back-end services for the most popular podcasts in our platform; each subset gets a dedicated SQLite database and are not created equally. The most famous podcasts (we assign a "popularity" score to each one based on sentiment, rankings on different pages like Youtube, Google, and Apple) we put in one database which is going to be used only for building pages at build-time.

Withing the subset you would find podcasts like: - The Joe Rogan Experience- Call Her Daddy - The Lex Fridman Podcast - The Tim Ferriss Show - The Huberman Lab - The Joe Budden Podcast - ALL-IN Podcast

And quite a few more (we have a trade-off here that the more podcasts we add here, the slower the build time will be, but the also the most popular podcasts with the more traffic will be exquisitely fast to enter.)