Connector FactoryDiscoverCreateShare
Connectors
DocsGitHub5

Pipeline scaffold

Below is a visualization of the pipeline registry scaffolds. Each section shows the folder layout generated by the scaffold and the intent of the key folders.

Meta scaffold

  • {pipeline}
    • _meta
      • README.md
      • pipeline.json
      • assets
        • from
          • README.md
        • to
          • README.md
    • {version}
      • _meta
        • README.md
        • version.json
        • assets
      • {author}
        • _meta
          • README.md
          • CHANGELOG.md
          • LICENSE
          • pipeline.json
          • assets
        • python
        • typescript

Python implementation scaffold

  • python
    • {implementation}
      • .gitignore
      • .env.example
      • README.md
      • pyproject.toml
      • docs
        • getting-started.md
        • configuration.md
        • outputs.md
      • scripts
        • lineage.py
      • schemas
        • index.json
      • lineage
        • schemas
          • index.json
          • relational
            • tables.json
            • README.md
          • files
            • manifest.json
            • README.md
      • moose
        • lineage.manifest.json
      • src
        • {packageName}
          • __init__.py
          • runner.py
          • config.py
          • extract
            • __init__.py
            • base_extractor.py
          • transform
            • __init__.py
            • base_transformer.py
          • load
            • __init__.py
            • base_loader.py
      • tests
        • test_runner.py
      • examples
        • basic_usage.py

TypeScript implementation scaffold

  • typescript
    • {implementation}
      • .gitignore
      • .env.example
      • README.md
      • package.json
      • tsconfig.json
      • vitest.config.ts
      • docs
        • getting-started.md
        • configuration.md
        • outputs.md
      • schemas
        • index.json
      • moose
        • lineage.manifest.json
      • lineage
        • schemas
          • index.json
          • relational
            • tables.json
            • README.md
          • files
            • manifest.json
            • README.md
      • src
        • index.ts
        • runner.ts
        • config.ts
        • extract
          • baseExtractor.ts
        • transform
          • baseTransformer.ts
        • load
          • baseLoader.ts
      • scripts
        • lineage.mjs
      • tests
        • runner.test.ts
      • examples
        • basic-usage.ts

What the folders mean

{pipeline}/_meta

  • Holds pipeline-level metadata that is provider-agnostic.
  • Files: pipeline.json, README.md, and assets/ for images and lineage diagrams.
  • Avoid placing detailed implementation docs here; keep docs near the language implementation.

{pipeline}/{version}/_meta

  • Version-level metadata for the pipeline.
  • Files: version.json, README.md, and assets/.

{pipeline}/{version}/{author}/_meta

  • Provider/author-specific metadata and lineage assets.
  • Files: pipeline.json (executable spec), CHANGELOG.md, LICENSE, assets/ (e.g., lineage.mmd, lineage.svg).

Language implementations under {pipeline}/{version}/{author}

  • python/{implementation}/ and typescript/{implementation}/ contain helpers and runnable code.
  • Prefer placing docs adjacent to the implementation:
    • docs/ for human-facing guides (getting started, config, outputs)
    • schemas/ at the top level of the language directory for machine-readable datasets/index
    • src/ for code (with subfolders like extract/, transform/, load/)
    • tests/ for unit tests
    • scripts/ for automation like lineage generation

Notes

  • _meta folders are for metadata and assets. Keep implementation docs inside the language directories.
  • Place schemas at the top level of each language implementation (not under src).