Published in Towards Data Science·PinnedMember-onlyData Integration Strategies for Time Series DatabasesExploring popular data integration strategies for TSDBs including ETL, ELT, and CDC — As digital transformation reaches more industries, the number of data points generated is growing exponentially. As such, data integration strategies to collect such large volumes of data from different sources in varying formats and structures are now a primary concern for data engineering teams. …Time Series Database6 min readTime Series Database6 min read
Published in Level Up Coding·PinnedMember-onlyChange Data Capture with QuestDB & DebeziumStreaming data into QuestDB with change data capture via Debezium and Kafka Connect. Modern data architecture has largely shifted away from the ETL (Extract-Transform-Load) paradigm to ELT (Extract-Load-Transform) where raw data is first loaded into a data lake before transformations are applied (e.g., aggregations, joins) for further analysis. Traditional ETL…Timeseries11 min readTimeseries11 min read
Published in Geek Culture·PinnedMember-onlyA Practical Guide to Improving the Developer Experience with Kubernetes at StartupsReducing the “Kubernetes tax” for startups without a dedicated platform engineering or developer experience team. Let’s face it. Kubernetes is still not easy in 2023. Despite years of development and innovation, Kubernetes remains a challenge for small teams. For non-infrastructure engineers not living and breathing Kubernetes every day, there is…Kubernetes5 min readKubernetes5 min read
Published in Level Up Coding·PinnedMember-onlyLessons Learned from Upgrading to PostgreSQL 13Documenting performance improvements with VACUUM, ANALYZE, and REINDEX as well as some AWS RDS surprises. With the PostgreSQL community planning to deprecate support for PostgreSQL 10 after November 10, 2022 and AWS RDS following suit in April 17, 2023, I was recently tasked with upgrading our database to PostgreSQL 13…AWS4 min readAWS4 min read
Published in ITNEXT·PinnedMember-onlyHow to Design and Provision a Production-Ready EKS ClusterA comprehensive guide to create and configure a production-grade Kubernetes cluster on AWS with Terraform, Helm, and other open-source tools. According to Flexera’s 2021 State of Cloud report, AWS leads the container orchestration market share with 51% of the respondents using Amazon EKS and ECS compared to 43% for AKS…Kubernetes8 min readKubernetes8 min read
Published in Dev Genius·Jan 27Member-onlyDeploying HarperDB on AWS and GCPDiving into a distributed database platform designed for the edge on AWS and GCP. I recently came across an interesting database called HarperDB that aims to simplify most database operations that developers have trouble with while maintaining high performance (e.g., changing schemas over time, performance degradation with multi-region, etc). HarperDB…Database7 min readDatabase7 min read
Published in Dev Genius·Jan 10Member-onlyIntroducing the QuestDB Kafka ConnectorA more performant change data capture (CDC) setup to stream data from Kafka to QuestDB leveraging the InfluxDB line protocol. Back in 2021, I wrote about a reference architecture for streaming realtime cryptocurrency prices with Kafka and QuestDB. That project polled the public Coinbase API for the price of various…Questdb4 min readQuestdb4 min read
Published in Dev Genius·Nov 1, 2022Member-onlyData Lifecycle with QuestDBFor most applications dealing with time series data, the value of each data point diminishes over time as the granularity of the dataset loses relevance as it gets stale. For example, when applying a real-time anomaly detection model, more granular data (e.g., data collected at second resolution), would yield better…Data Science5 min readData Science5 min read
Published in Better Programming·Jun 13, 2022Member-onlySimplifying Kubernetes CI/CD With DevtronHow to allow developers to implement an effective CI/CD pipeline without dealing with YAML — When organizations think of fostering a DevOps culture, building out an effective continuous integration (CI) and continuous deployment (CD) pipeline is usually brought up as the first step to success. Nowadays, infrastructure teams have a plethora of both open-source and licensed tools such as Jenkins, CircleCI, Github Actions, and ArgoCD…Kubernetes5 min readKubernetes5 min read
Published in ITNEXT·Apr 27, 2022Member-onlyEKS Blueprints: IaC Modules for Production-Ready KubernetesBootstrap a fully-featured Kubernetes cluster on AWS with EKS Blueprints via Terraform and AWS CDK. Earlier this week, AWS team announced EKS Blueprints, a collection of open-source, infrastructure as code (IaC) modules to bootstrap an EKS cluster with popular Kubernetes tools such as Prometheus, Karpenter, Nginx, Traefik, AWS Load Balancer…AWS4 min readAWS4 min read
