Sinks

Sinks are plugins that receive Records and push them to destination systems. Each Record contains an Entity (with a flat properties map) and a list of Edges representing relationships such as ownership and lineage.

To use a sink, add a sinks block to your recipe:

sinks:
  - name: compass
    config:
      host: https://compass.example.com

Supported Sinks

Sink	Description	Output
`compass`	Send entities and edges to Compass	Compass API (HTTP)
`kafka`	Publish entity as protobuf to a Kafka topic	Kafka topic
`file`	Write records to a local file	NDJSON or YAML file
`console`	Print records to stdout	Standard output
`http`	Send entity JSON to any HTTP endpoint	HTTP API
`stencil`	Register table schemas in Stencil	Stencil API (HTTP)
`gcs`	Write records as NDJSON to Google Cloud Storage	GCS bucket
`s3`	Write records as NDJSON to Amazon S3 or S3-compatible storage	S3 bucket
`azure_blob`	Write records as NDJSON to Azure Blob Storage	Azure container

Compass

Sends each Record to Compass via HTTP. The entity is upserted with its properties, and all edges are upserted uniformly via the UpsertEdge endpoint.

sinks:
  - name: compass
    config:
      host: https://compass.example.com
      headers:
        Compass-User-UUID: meteor@raystack.io

Key	Description	Required
`host`	Compass service hostname	Yes
`headers`	Additional HTTP headers (comma-separated values for multiple)	No

Kafka

Serializes the entity as a protobuf message and publishes it to a Kafka topic. The optional key_path extracts a field from the entity to use as the Kafka message key.

sinks:
  - name: kafka
    config:
      brokers: localhost:9092
      topic: metadata-topic
      key_path: .Urn

Key	Description	Required
`brokers`	Comma-separated list of Kafka broker addresses	Yes
`topic`	Kafka topic to publish messages to	Yes
`key_path`	Field path on the entity proto to use as the message key (e.g. `.Urn`)	No

File

Writes records to a local file in NDJSON or YAML format. Each record is serialized as JSON (entity + edges).

sinks:
  - name: file
    config:
      path: ./output.ndjson
      format: ndjson
      overwrite: true

Key	Description	Required
`path`	Output file path	Yes
`format`	Output format: `ndjson` or `yaml`	Yes
`overwrite`	Overwrite existing file (default `true`)	No

Console

Prints each record as JSON to stdout. Useful for debugging recipes.

sinks:
  - name: console

No configuration required.

HTTP

Sends the entity as JSON to an arbitrary HTTP endpoint. The URL supports Go template variables from the entity (e.g. {{ .Type }}, {{ .Urn }}). An optional Tengo script can transform the payload before sending.

sinks:
  - name: http
    config:
      url: https://example.com/metadata/{{ .Type }}
      method: PUT
      success_code: 200
      headers:
        Authorization: Bearer token

Key	Description	Required
`url`	Target URL (supports Go template variables)	Yes
`method`	HTTP method (`GET`, `POST`, `PUT`, `PATCH`, etc.)	Yes
`success_code`	Expected HTTP status code for success (default `200`)	No
`headers`	Additional HTTP headers	No
`script.engine`	Script engine for payload transformation (`tengo`)	No
`script.source`	Tengo script source code	No

Stencil

Registers table column schemas in Stencil as JSON Schema or Avro. Only entities with a columns field in their properties are processed. Column types from BigQuery and PostgreSQL are automatically mapped to the target schema format.

sinks:
  - name: stencil
    config:
      host: https://stencil.example.com
      namespace_id: myNamespace
      format: json

Key	Description	Required
`host`	Stencil service hostname	Yes
`namespace_id`	Stencil namespace to register schemas under	Yes
`format`	Schema format: `json` or `avro` (default `json`)	No

GCS

Writes records as NDJSON to a Google Cloud Storage bucket. Each record is serialized as a JSON line. The output object is named with an optional prefix and a timestamp.

sinks:
  - name: gcs
    config:
      project_id: my-gcp-project
      url: gcs://bucket_name/target_folder
      object_prefix: github-users
      service_account_base64: <base64-encoded-service-account-key>

Key	Description	Required
`project_id`	GCP project ID	Yes
`url`	GCS destination in the form `gcs://bucket/path`	Yes
`object_prefix`	Prefix for the output object name	No
`service_account_base64`	Base64-encoded service account JSON key	No*
`service_account_json`	Service account JSON key as a string	No*

*One of service_account_base64 or service_account_json is required.

S3

Writes records as NDJSON to an Amazon S3 bucket or any S3-compatible storage (MinIO, DigitalOcean Spaces, etc.). Each record is serialized as a JSON line. The output object is named with an optional prefix and a timestamp.

sinks:
  - name: s3
    config:
      bucket_url: s3://my-bucket/metadata
      region: us-east-1
      object_prefix: github-users
      access_key_id: AKIAIOSFODNN7EXAMPLE
      secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Key	Description	Required
`bucket_url`	S3 destination in the form `s3://bucket/path`	Yes
`region`	AWS region	Yes
`object_prefix`	Prefix for the output object name	No
`access_key_id`	AWS access key ID	No*
`secret_access_key`	AWS secret access key	No*
`endpoint`	Custom S3 endpoint for S3-compatible stores	No

*If credentials are omitted, the default AWS credential chain is used (env vars, instance profile, etc.).

Azure Blob

Writes records as NDJSON to an Azure Blob Storage container. Each record is serialized as a JSON line. The output blob is named with an optional prefix and a timestamp.

sinks:
  - name: azure_blob
    config:
      storage_account_url: https://myaccount.blob.core.windows.net
      container_name: my-container
      object_prefix: github-users
      account_key: <account-key>

Key	Description	Required
`storage_account_url`	Azure storage account URL	Yes
`container_name`	Blob container name	Yes
`object_prefix`	Prefix for the output blob name	No
`account_key`	Azure storage account key	No*
`connection_string`	Azure storage connection string	No*

*One of account_key or connection_string is required.