Achieving strong isolation without sacrificing resource utilization with Virtual Clusters. Kubernetes is eating the cloud-native world. In the most recent Cloud Native Computing Foundation (CNCF) Annual Survey, 96% of organizations reported being either using or evaluating Kubernetes, up from 83% in 2020 and 78% in 2019. It is also estimated…

Exploring popular data integration strategies for TSDBs including ETL, ELT, and CDC — As digital transformation reaches more industries, the number of data points generated is growing exponentially. As such, data integration strategies to collect such large volumes of data from different sources in varying formats and structures are now a primary concern for data engineering teams. …

Reducing the “Kubernetes tax” for startups without a dedicated platform engineering or developer experience team. Let’s face it. Kubernetes is still not easy in 2023. Despite years of development and innovation, Kubernetes remains a challenge for small teams. For non-infrastructure engineers not living and breathing Kubernetes every day, there is…

A comprehensive guide to create and configure a production-grade Kubernetes cluster on AWS with Terraform, Helm, and other open-source tools. According to Flexera’s 2021 State of Cloud report, AWS leads the container orchestration market share with 51% of the respondents using Amazon EKS and ECS compared to 43% for AKS…

Diving into a distributed database platform designed for the edge on AWS and GCP. I recently came across an interesting database called HarperDB that aims to simplify most database operations that developers have trouble with while maintaining high performance (e.g., changing schemas over time, performance degradation with multi-region, etc). HarperDB…

A more performant change data capture (CDC) setup to stream data from Kafka to QuestDB leveraging the InfluxDB line protocol. Back in 2021, I wrote about a reference architecture for streaming realtime cryptocurrency prices with Kafka and QuestDB. That project polled the public Coinbase API for the price of various…

Streaming data into QuestDB with change data capture via Debezium and Kafka Connect. Modern data architecture has largely shifted away from the ETL (Extract-Transform-Load) paradigm to ELT (Extract-Load-Transform) where raw data is first loaded into a data lake before transformations are applied (e.g., aggregations, joins) for further analysis. Traditional ETL…

For most applications dealing with time series data, the value of each data point diminishes over time as the granularity of the dataset loses relevance as it gets stale. For example, when applying a real-time anomaly detection model, more granular data (e.g., data collected at second resolution), would yield better…

Documenting performance improvements with VACUUM, ANALYZE, and REINDEX as well as some AWS RDS surprises. With the PostgreSQL community planning to deprecate support for PostgreSQL 10 after November 10, 2022 and AWS RDS following suit in April 17, 2023, I was recently tasked with upgrading our database to PostgreSQL 13…