Fixing Checkpoint Recovery in SeaTunnel’s BigQuery Connector

While working on the BigQuery Sink Connector for Apache SeaTunnel, I initially thought the main challenge would be straightforward: write rows into BigQuery correctly.

But as the implementation evolved, I realized the real problem was deeper than just calling the BigQuery API. The hard part was aligning BigQuery’s external write visibility model with SeaTunnel’s checkpoint and restore lifecycle.

This post summarizes the design issue I encountered while using the BigQuery Storage Write API, why the initial pending-stream design was not strong enough for checkpoint recovery, and why I eventually redesigned the batch write path around buffered streams.

Initial Design: Pending Streams for Batch Writes

BigQuery Storage Write API provides several write stream types. At first, Pending Stream looked like a natural fit for batch writes.

The flow is roughly:

Create PENDING streamAppendRowsFinalizeWriteStreamBatchCommitWriteStreams

Data written to a pending stream is not visible to readers until BatchCommitWriteStreams is called. Since BigQuery can...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE