A Cloud Data Fusion pipeline to upload audit records to BigQuery

Cloud SQL for SQL Server offers configurable auditing, capturing events like login attempts, DDLs, and DMLs. These events are stored locally on the instance for up to 7 days and can be preserved longer in GCS buckets.

Some customers want to access and analyze their Cloud SQL for SQL Server audit data in other systems, like BigQuery or Tableau. While a custom-built pipeline can achieve this, it introduces complexity and maintenance burdens. Constant investment is needed for support, monitoring, and ensuring no audit records are lost.

In this blog, we walk through the steps to build a flexible pipeline to output audit records to internal or external sinks with minimal coding.

Prerequisites

Cloud Data Fusion doesn't directly support private IP Cloud SQL for SQL Server instances. To connect to these, you'll need to set up a Cloud SQL Auth Proxy as a bridge between Cloud Data Fusion and ...