Meteor Metadata Model

Meteor uses an Entity + Edge model to represent metadata. Each extractor emits one or more Records, where each Record contains an Entity and zero or more Edges.

Entity

An Entity represents a metadata resource (table, dashboard, topic, job, user, etc.). All entity types share a single flat structure:

There are no separate typed schemas (e.g. no Table, Dashboard, Bucket proto types). All metadata is stored as flat key-value pairs in properties.

Edge

An Edge represents a relationship between two entities (ownership, lineage, etc.):

Relationship Types

• owned_by: Indicates ownership. Replaces the old owners field.
• derived_from: Indicates an upstream dependency — the entity depends on or reads from the target.
• generates: Indicates a downstream output — the entity produces or writes to the target.
• member_of: Indicates membership in a group, team, or organisation.
• belongs_to: Indicates an entity belongs to a parent entity (e.g. document to repository).
• child_of: Indicates a parent-child hierarchy (e.g. sub-page within a page).
• consumed_by: Indicates a consumer group reads from a topic.

Record

A Record is the unit of data flowing through the Meteor pipeline. It wraps an Entity and its associated Edges:

• record.Entity() returns the Entity.
• record.Edges() returns the list of Edges.

Supported Entity Types

• api: API schemas from OpenAPI specs and protobuf definitions
• application: Services communicating over APIs
• bucket: Cloud storage containers (GCS, etc.)
• consumer_group: Kafka consumer groups
• dashboard: Data visualization dashboards (Metabase, Grafana, Tableau, etc.)
• document: Documentation pages (Confluence, Notion, GitHub, etc.)
• job: Scheduled/recurring data transformation tasks (Optimus, etc.)
• model: Machine learning models
• repository: Source code repositories (GitHub, etc.)
• table: Database tables and views (BigQuery, Postgres, MySQL, etc.)
• team: Teams within an organisation (GitHub, etc.)
• topic: Message bus topics (Kafka, etc.)
• user: User accounts (GitHub, Google Suite, etc.)

The HTTP extractor can emit custom entity types as defined by its script configuration.

To see which extractors emit which entity types, run meteor entities or visit here.

Usage

import (
    "github.com/raystack/meteor/models"
    meteorv1beta1 "github.com/raystack/meteor/models/raystack/meteor/v1beta1"
    "google.golang.org/protobuf/types/known/structpb"
)

func main() {
    // Build properties
    props, _ := structpb.NewStruct(map[string]interface{}{
        "schema": map[string]interface{}{
            "columns": []interface{}{
                map[string]interface{}{
                    "name":        "column_name",
                    "data_type":   "varchar",
                    "is_nullable": true,
                    "length":      256,
                },
            },
        },
    })

    // Create an Entity
    entity := &meteorv1beta1.Entity{
        Urn:         "urn:postgres:mydb:table:mydb.my_table",
        Type:        "table",
        Name:        "my_table",
        Source:      "postgres",
        Properties:  props,
    }

    // Create ownership and lineage as Edges
    edges := []*meteorv1beta1.Edge{
        {
            SourceUrn: "urn:postgres:mydb:table:mydb.my_table",
            TargetUrn: "urn:user:myorg:user:alice",
            Type:      "owned_by",
            Source:    "postgres",
        },
    }

    // Wrap in a Record for the pipeline
    record := models.NewRecord(entity, edges)
    _ = record
}