Configuration
Meteor is configured via a YAML file (default: ./meteor.yaml) and environment variables. This page covers all available settings.
Agent Configuration
These settings control Meteor's runtime behavior and can be set in meteor.yaml or as environment variables.
LOG_LEVEL
- Example value:
debug - Type:
optional - Default:
info - Controls log verbosity. Supported values:
debug,info,warn,error. - Can also be overridden per-run with
meteor run --log-level debug.
MAX_RETRIES
- Example value:
3 - Type:
optional - Default:
5 - Maximum number of retry attempts for extractor and sink operations.
RETRY_INITIAL_INTERVAL_SECONDS
- Example value:
10 - Type:
optional - Default:
5 - Initial interval in seconds between retry attempts. Uses exponential backoff.
STOP_ON_SINK_ERROR
- Example value:
true - Type:
optional - Default:
false - When
true, stops the pipeline if any sink returns an error after exhausting retries. Whenfalse, logs the error and continues.
SINK_BATCH_SIZE
- Example value:
10 - Type:
optional - Default:
1 - Number of records to batch before sending to each sink.
OTEL_ENABLED
- Example value:
true - Type:
optional - Default:
false - Enable OpenTelemetry metrics and tracing.
OTEL_COLLECTOR_ADDR
- Example value:
otel-collector:4317 - Type:
optional - Default:
localhost:4317 - Address of the OpenTelemetry collector (gRPC endpoint).
OTEL_TRACE_SAMPLE_PROBABILITY
- Example value:
0.5 - Type:
optional - Default:
1 - Trace sampling probability (0.0 to 1.0).
Sample meteor.yaml
LOG_LEVEL: info
MAX_RETRIES: 5
RETRY_INITIAL_INTERVAL_SECONDS: 5
STOP_ON_SINK_ERROR: false
SINK_BATCH_SIZE: 1Environment Variables
Meteor reads environment variables with the METEOR_ prefix as template data for recipes. This allows you to inject credentials and configuration without hardcoding them.
For example, setting METEOR_BIGQUERY_PROJECT_ID=my-project makes bigquery_project_id available in recipes:
source:
name: bigquery
config:
project_id: "{{ .bigquery_project_id }}"See Dynamic recipe value for details.
Extractor Configuration
Each extractor has its own configuration schema. Common patterns:
Database Extractors
Most database extractors (postgres, mysql, mssql, mariadb, oracle, clickhouse, redshift, snowflake, presto) accept:
| Key | Description |
|---|---|
connection_url | Database connection string (format varies by database) |
exclude | Map of databases/schemas/tables to exclude from extraction |
Cloud Service Extractors
Extractors for GCP services (bigquery, bigtable, gcs) commonly accept:
| Key | Description |
|---|---|
project_id | GCP project ID |
service_account_base64 | Base64-encoded service account JSON key |
service_account_json | Service account JSON key (inline) |
API-based Extractors
Extractors for BI tools (tableau, metabase, superset, grafana, redash) commonly accept:
| Key | Description |
|---|---|
host | Service host URL |
username | Authentication username |
password | Authentication password |
For complete configuration details of each extractor, refer to the individual README files linked from the extractors reference.
Sink Configuration
Each sink has its own configuration. Common patterns:
| Sink | Key Config Fields |
|---|---|
compass | host, type, mapping |
kafka | brokers, topic, key_path |
http | method, url, headers, success_code |
gcs | project_id, url, object_prefix, service_account_base64 |
file | path, format |
stencil | host, namespace_id, schema_id, format |
For complete configuration details, refer to the sinks reference.
Processor Configuration
| Processor | Key Config Fields |
|---|---|
enrich | attributes — key-value map to merge into entity properties |
labels | labels — key-value map to append to entity properties labels |
script | engine, script — Tengo script for custom transformation |
For complete configuration details, refer to the processors reference.