Published in ITNEXT·PinnedMember-onlyKubernetes Sandbox Environments with Virtual ClustersAchieving strong isolation without sacrificing resource utilization with Virtual Clusters. Kubernetes is eating the cloud-native world. In the most recent Cloud Native Computing Foundation (CNCF) Annual Survey, 96% of organizations reported being either using or evaluating Kubernetes, up from 83% in 2020 and 78% in 2019. It is also estimated…Kubernetes5 min readKubernetes5 min read
Published in Towards Data Science·PinnedMember-onlyData Integration Strategies for Time Series DatabasesExploring popular data integration strategies for TSDBs including ETL, ELT, and CDC — As digital transformation reaches more industries, the number of data points generated is growing exponentially. As such, data integration strategies to collect such large volumes of data from different sources in varying formats and structures are now a primary concern for data engineering teams. …Time Series Database6 min readTime Series Database6 min read
Published in Geek Culture·PinnedMember-onlyA Practical Guide to Improving the Developer Experience with Kubernetes at StartupsReducing the “Kubernetes tax” for startups without a dedicated platform engineering or developer experience team. Let’s face it. Kubernetes is still not easy in 2023. Despite years of development and innovation, Kubernetes remains a challenge for small teams. For non-infrastructure engineers not living and breathing Kubernetes every day, there is…Kubernetes5 min readKubernetes5 min read
Published in Level Up Coding·PinnedMember-onlyLessons Learned from Upgrading to PostgreSQL 13Documenting performance improvements with VACUUM, ANALYZE, and REINDEX as well as some AWS RDS surprises. With the PostgreSQL community planning to deprecate support for PostgreSQL 10 after November 10, 2022 and AWS RDS following suit in April 17, 2023, I was recently tasked with upgrading our database to PostgreSQL 13…AWS4 min readAWS4 min read
Published in ITNEXT·PinnedMember-onlyHow to Design and Provision a Production-Ready EKS ClusterA comprehensive guide to create and configure a production-grade Kubernetes cluster on AWS with Terraform, Helm, and other open-source tools. According to Flexera’s 2021 State of Cloud report, AWS leads the container orchestration market share with 51% of the respondents using Amazon EKS and ECS compared to 43% for AKS…Kubernetes8 min readKubernetes8 min read
Published in ITNEXT·5 days agoMember-onlyMigrating from Relational Databases to Time Series DatabasesMore than 50 years after their inception, relational databases remain popular as ever to model, manage, and process data. In fact, 7 of the top 10 most popular database management systems are relational databases, including MySQL, SQL Server, and PostgreSQL. The rationale for the popularity is clear: There is a…Database6 min readDatabase6 min read
Published in Dev Genius·Jan 27Member-onlyDeploying HarperDB on AWS and GCPDiving into a distributed database platform designed for the edge on AWS and GCP. I recently came across an interesting database called HarperDB that aims to simplify most database operations that developers have trouble with while maintaining high performance (e.g., changing schemas over time, performance degradation with multi-region, etc). HarperDB…Database7 min readDatabase7 min read
Published in Dev Genius·Jan 10Member-onlyIntroducing the QuestDB Kafka ConnectorA more performant change data capture (CDC) setup to stream data from Kafka to QuestDB leveraging the InfluxDB line protocol. Back in 2021, I wrote about a reference architecture for streaming realtime cryptocurrency prices with Kafka and QuestDB. That project polled the public Coinbase API for the price of various…Questdb4 min readQuestdb4 min read
Published in Level Up Coding·Dec 26, 2022Member-onlyChange Data Capture with QuestDB & DebeziumStreaming data into QuestDB with change data capture via Debezium and Kafka Connect. Modern data architecture has largely shifted away from the ETL (Extract-Transform-Load) paradigm to ELT (Extract-Load-Transform) where raw data is first loaded into a data lake before transformations are applied (e.g., aggregations, joins) for further analysis. Traditional ETL…Timeseries11 min readTimeseries11 min read
Published in Dev Genius·Nov 1, 2022Member-onlyData Lifecycle with QuestDBFor most applications dealing with time series data, the value of each data point diminishes over time as the granularity of the dataset loses relevance as it gets stale. For example, when applying a real-time anomaly detection model, more granular data (e.g., data collected at second resolution), would yield better…Data Science5 min readData Science5 min read
