Cloudflare has embedded SQL-based log transformation into its Logpush export workflow. Transformers allow teams to filter, redact sensitive data, and reshape records before they leave Cloudflare, with outputs delivered in NDJSON format to security platforms, analytics tools, or object storage.
- Filters can remove health checks or internal data
- Redactions can conceal sensitive fields
- Reshaping can normalize schemas for downstream dashboards
AVAILABILITY AND SCOPE
Transformers are now generally available. Cloudflare notes that Logpush and Transformers were part of eight major updates to Cloudflare Observability, and that Logpush is available on all self-serve plans. TechStaged has also covered GitLab releases critical patch set: 19.4.2, 19.3.4, 19.2.8 address bugs and security fixes.
During Birthday Week, Cloudflare extended access to Logpush and Transformers to a broader set of customers.
HOW TRANSFORMERS WORK INSIDE LOGPUSH
Transformers run directly in the Logpush delivery path (Bufstream), applying a SQL query to each batch of records before export. A Transformer is defined against a Logpush dataset and can rename, nest, or remove fields while adding new information such as provider and error context. The transformed output is then exported as NDJSON to destinations like SIEMs, analytics platforms, or object storage.
- A Transformer is a SQL query written for a specific dataset
- Queries are executed in-process to avoid remote calls and maintain throughput
- Cloudflare uses Apache DataFusion (Rust) and Apache Arrow for fast, columnar processing
PERFORMANCE AND ARCHITECTURE
Cloudflare’s data pipeline handles billions of events per second. In testing, a prepared Transformer operating on a synthetic 5,000-row http_requests batch, run from a single Go goroutine, completed in about 3 milliseconds (roughly 1.6 million rows per second).
To minimize cross-language friction, DataFusion runs with Arrow IPC for efficient data exchange between Go and Rust, and transforms are performed on Arrow RecordBatches before being converted to NDJSON for export.
- Go/Rust boundary managed via Arrow IPC
- Batches are decompressed, parsed into typed RecordBatches, and fed to DataFusion
- Results are re-packaged into NDJSON and uploaded to the destination
GETTING STARTED WITH TRANSFORMER STUDIO
Users can create, preview, and manage Transformers in Transformer Studio or via API. After saving, the Transformer is versioned, enabling review of changes over time.
To attach a Transformer to a Logpush job, choose the dataset, write the SQL, preview against sample records, and link to a compatible NDJSON Logpush job. Transformers are available on Free, Pro, Business, and Enterprise plans, with pricing details in the Transformer and Logpush pricing documentation.
- Open Transformer Studio from the Logpush page
- Attach to a matching Logpush job
- Review versions and changes over time
RELATED COVERAGE
- GitLab releases critical patch set: 19.4.2, 19.3.4, 19.2.8 address bugs and security fixes
- JetBrains launches Build Your Game Challenge for JetBrains Academy community, with course-based entry and $50 jury prizes
- Cloudflare introduces on-demand CPU and memory profiling with flamegraphs for Workers and Durable Objects
- GitHub Copilot for JetBrains adds admin controls, clearer diagnostics and improved chat navigation
- Developer Tools articles
SOURCES
- Cloudflare Blog: Filter, reshape, and redact logs before export with Transformers Published · Primary source
