The Great Un-monolithing

The Post-Gazette switched from a home-build publishing stack and CMS (called Bridge) to Libercus in 2013. After some data migrations, templates, undocumented APIs, visits from CEOs, and a bit of workaholicism, it was finally done. Except not: Libercus was pretty terrible once you had any more than a handful of storylists on a single page. The homepage had a ton. So we decided, just a few years later: let's ditch Libercus as a frontend. We would still use it as a CMS and source of truth, but it would no longer be our publishing stack.

After quite a bit of tinkering, learning, and plenty of ice cream sandwiches, our new publishing stack was born.

Launching this...thing, was very tough. We had a barely functioning Cloudera server. We had 3 web servers that had to run background tasks, serve HTML and also run nginx and varnish. We had a system that could take up to 15 minutes for a story to show up after publishing. We had an unhappy newsroom. But they dealt with it, all the way until 2024.

The Post-Gazette newsroom had patience I could only dream of. Occasional issues would come up, but they'd deal with it with grace and acceptance. But I had had enough. I was tired of having to wake up at 4AM to fix an issue. I was tired of 50% of my day being spent fixing bugs. I built a broken system, and I needed to fix it.

Around this time, our systems team got kind of taken over by the shared services team at BCI. There were some growing pains, but they were also willing to expand us. My initial pitch was 3 web servers, 3 Varnish servers, 2 data servers. We'd keep Cloudera. We'd separate the web logic from the caching logic. It would be better, but would it really be great? So I started pursuing docker, containerization, trying to find ways to get things more modern. I was struggling building an API gateway, an orchestrator, etc. Tried docker swarm and wasn't a fan, and finally stumbled upon Kubernetes. It seemed to have everything I need as a solo dev: redundancy, self healing, easy deployment. The dream of no more sleepless nights was too good to pass up, so I went for it. I sent my request.

Is that actually better? Who knows

The rule that made 29 services survivable for one keeper: every service looks the same. Same repo layout, same base image, same health checks, same way onto and off the bus. The novelty budget goes to the problem, not the plumbing.

# Project: archivist
# File: main.py
# Description: Main entry point for the archivist service.

import os
import uvicorn  # type: ignore
from app.archivist import app
from app.logging.config import logconfig_dict
from config import Settings  # Changed from app.config import Settings

if __name__ == "__main__":
    settings = Settings()

    uvicorn.run(
        app,
        host=settings.HOST,
        port=settings.PORT,
        log_config=logconfig_dict
    )

Every service gets an entry point, a main.py file that initializes it and starts the (mostly) FastAPI process. Every service gets an API, to check the health and for prometheus metrics. Every service scaffolded the same, same log output, same general structure throughout. Interface first and SOLID led the way.

# Description: MongoDB data access layer for story data.
# Standard imports
import datetime
import logging
from typing import Any, List, Optional, Tuple, Dict

# External imports
from motor.motor_asyncio import AsyncIOMotorClient  # type: ignore
from bson.objectid import ObjectId  # type: ignore
from bson.errors import InvalidId  # type: ignore
from pymongo import ReadPreference  # type: ignore
from pymongo.errors import ServerSelectionTimeoutError, NetworkTimeout  # type: ignore

# Internal imports
from interfaces import StoryDataInterface
from app.domain_model import QueryParams, FilterCondition, SortCondition, Operator, RegexOptions

logger = logging.getLogger(__name__)


class MongoStoryData(StoryDataInterface):
    STORY_COLLECTION = "stories"

    # Class-level counter for concurrent aggregations - per instance
    _concurrent_aggregations = 0
    _MAX_CONCURRENT_AGGREGATIONS = 10  # Reduced from 10 to account for multiple workers

    #
    # Initialization and lifecycle methods
    #
    def __init__(self, db_url: str, db_name: str, read_preference: str = "secondaryPreferred"):
        """
        Initializes the MongoDB client and database connection.
        Args:
            db_url (str): The MongoDB connection URL.
            db_name (str): The name of the database to connect to.
            read_preference (str): The read preference for the MongoDB client.
                - "primary" for primary node
                - "secondary" for secondary node
                - "primaryPreferred" for primary node if available, otherwise secondary
                - "secondaryPreferred" for secondary node if available, otherwise primary
                - "nearest" for the nearest node
        """
        read_pref_map = {
            "primary": ReadPreference.PRIMARY,
            "secondary": ReadPreference.SECONDARY,
            "primaryPreferred": ReadPreference.PRIMARY_PREFERRED,
            "secondaryPreferred": ReadPreference.SECONDARY_PREFERRED,
            "nearest": ReadPreference.NEAREST
        }

        mongo_read_pref = read_pref_map.get(read_preference, ReadPreference.SECONDARY_PREFERRED)

        self.client = AsyncIOMotorClient(
            db_url,
            read_preference=mongo_read_pref,
            serverSelectionTimeoutMS=3000,
            connectTimeoutMS=3000,
            socketTimeoutMS=5000,
        )
        self.db = self.client[db_name]
        self.collection = self.db[self.STORY_COLLECTION]

        self._read_pref = mongo_read_pref

To migrate without interruption to our digital platform, I had to create this cluster while keeping the old servers running. This resulted in a lot of overtime (not recommended), a lot of working late into the night, a lot of missed time with my kid, regrettably. I was able to create the working microservices cluster, mirroring our website. Before I was ready to cutover, things started breaking. Feeds on the old platform were breaking, not working correctly. The infinite scrolls were no longer infinite. I had to adapt, so I started migrating parts of it one at a time. First, routing some of the story lists to the new cluster.

function pg_mostRecent($section, $page)
{
    $app = \Slim\Slim::getInstance();
    $app->contentType("application/json");

    if ("local/breaking" == $section) {
        $section = "all";
    }

    // Use milize.toledoblade.com/recent/1/$section/$page/
    $dataURL = "http://milize.toledoblade.com/recent/1/{$section}/{$page}/";
    $data = @file_get_contents($dataURL);

    $dataDecoded = json_decode($data, true);
    if (isset($dataDecoded["status"])) {
        if ("ok" === $dataDecoded["status"]) {
            echo $data;
            exit;
        }
    }

    $appName    = "recent";
    $appSection = "";
    $apiName    = $section;

    $decoder = new RetrieveJSON($appName, $appSection, $apiName, $page);
    $time = $decoder->getLastMod();
    $app->lastModified($decoder->getLastMod());
    header("Last-Modified: " . $app->response->headers->get("Last-Modified"));
    if ($time === strtotime($app->request->headers->get('IF_MODIFIED_SINCE'))) {
        http_response_code(304);
        exit;
    }

    $decoder->getAndDecodePage();
    $data = $decoder->getContents();

    echo $data;
}

Finally, the whole cluster was eventually cutover, and the results...well, they were...

404s, stories missing (I am aware that's the same thing), sections not updating properly, sections with the wrong content, a random "Oh, we're moving the cluster to Ohio, you gotta do this all again".

Learning Kubernetes in 6 months was tough. There was a crazy learning curve, and even a year later, I'm not sure I still understand all of it. Certain services went crazy (Anbu blew up the cluster a few times, our cache invalidator and primer), certain services were undertuned, certain services weren't even my fault (looking at you Jenkins and your updates that break 80% of the plugins). But the newsroom stuck with me, patient as they were, and we reached stability.