How I Replaced Legacy SSIS with Real-Time GCP Pipelines
How I Replaced Legacy SSIS with Real-Time Data Pipelines (and Saved Costs!) A few years into working with SQL Server Integration Services (SSIS), I hit the same wall a lot of data teams eventually hit: nightly batch jobs that took longer every month, a server that needed constant babysitting, and a growing list of "just run it again manually" incidents. This is the story of moving that pipeline to a GCP-native setup — what worked, what I underestimated, and what it actually saved. 📘 The Starting Point The original setup was a fairly typical on-prem pattern: Source SQL Server → SSIS nightly ETL job → Data Warehouse (on-prem) It worked — until data volume grew. The nightly job window kept creeping later, sometimes bleeding into business hours, and every schema change at the source meant manually re-mapping columns in the SSIS designer. 🔧 The Migration Path Rather than a big-bang rewrite, I moved piece by piece: Google Cloud Datastream replaced the extraction step...