Pipelines

Execute batch operations from YAML or JSON pipeline files for complex automation workflows.

Overview

Pipelines allow you to:

Define multi-step workflows in declarative YAML or JSON files
Use variable substitution for dynamic values
Handle errors gracefully with continue-on-error
Validate pipelines before execution
Preview with dry-run mode

Pipeline Structure

A pipeline file has four fields:

name: My Pipeline            # Required: pipeline name
description: What it does    # Optional: description
variables:                   # Optional: key-value pairs for substitution
  BUCKET: my-bucket
  PROJECT_ID: "12345"
steps:                       # Required: list of steps to execute
  - name: Step Name          # Required: step display name
    command: bucket list     # Required: raps subcommand to run
    continue_on_error: false # Optional: don't abort on failure (default: false)
    condition: "true"        # Optional: skip if falsy (empty, "false", or "0")

Each step’s command is a raps subcommand string (without the raps prefix). For example, bucket list, object upload my-bucket -f model.rvt, or auth test.

Running Pipelines

# Execute a pipeline
raps pipeline run pipeline.yaml

# Dry-run (show commands without executing)
raps pipeline run pipeline.yaml --dry-run

# Continue past failures
raps pipeline run pipeline.yaml --continue-on-error

# Read pipeline from stdin (parsed as YAML)
cat pipeline.yaml | raps pipeline run -

Variable Substitution

Define variables at the top of your pipeline and reference them in step commands using ${VARIABLE} syntax:

name: Upload Pipeline
variables:
  BUCKET: my-project-bucket
  MODEL: building.rvt

steps:
  - name: Create Bucket
    command: "bucket create -k ${BUCKET} -p transient -r US"
    continue_on_error: true

  - name: Upload Model
    command: "object upload ${BUCKET} -f ./models/${MODEL}"

  - name: List Objects
    command: "object list ${BUCKET}"

Both ${VAR} and $VAR syntax are supported.

Conditional Steps

Steps can include a condition field. If the condition evaluates to falsy (empty string, "false", or "0"), the step is skipped:

steps:
  - name: Cleanup Old Data
    command: "bucket delete old-bucket -y"
    condition: "true"          # This step runs

  - name: Skip This
    command: "bucket list"
    condition: "false"         # This step is skipped

  - name: Also Skipped
    command: "bucket list"
    condition: "0"             # This step is skipped

Error Handling

By default, the pipeline aborts when a step fails. Use continue_on_error on individual steps or the --continue-on-error CLI flag to continue past failures:

steps:
  - name: Delete Old Bucket
    command: "bucket delete old-bucket -y"
    continue_on_error: true   # Pipeline continues if this fails

  - name: Create New Bucket
    command: "bucket create -k new-bucket -p persistent -r US"
    # Pipeline aborts if this fails (default behavior)

Validating Pipelines

Check a pipeline file for syntax errors before running it:

raps pipeline validate pipeline.yaml

Generating a Sample Pipeline

Generate a starter pipeline file:

# Generate YAML (default)
raps pipeline sample

# Generate to a specific file
raps pipeline sample --out-file my-pipeline.yaml

Complete Example

A practical pipeline for bucket lifecycle management:

name: Bucket Lifecycle
description: Create a bucket, upload files, and clean up

variables:
  BUCKET: raps-demo-bucket
  PROJECT_ID: "12345"

steps:
  - name: Verify Authentication
    command: auth test

  - name: List Existing Buckets
    command: bucket list

  - name: Create Project Bucket
    command: "bucket create -k ${BUCKET} -p transient -r US"
    continue_on_error: true

  - name: Upload Model
    command: "object upload ${BUCKET} -f ./models/building.rvt"

  - name: List Objects
    command: "object list ${BUCKET}"

  - name: Delete Bucket
    command: "bucket delete ${BUCKET} -y"
    continue_on_error: true

Pipeline Output

When a pipeline completes, you get a summary:

Pipeline: Bucket Lifecycle
  Create a bucket, upload files, and clean up
────────────────────────────────────────────────────────────

[1/6] Verify Authentication
  Command: auth test
  ✓ Success

[2/6] List Existing Buckets
  Command: bucket list
  ✓ Success

...

────────────────────────────────────────────────────────────
Pipeline Summary:
  ✓ 5 passed, ✗ 1 failed, → 0 skipped

For non-table output formats (JSON, YAML, CSV), the summary is machine-readable:

raps pipeline run pipeline.yaml --output json

Best Practices

Use descriptive step names — Makes logs and summaries easier to read
Set continue_on_error wisely — Only for non-critical steps like cleanup
Use dry-run first — Validate before executing: --dry-run
Store secrets in environment — Never put credentials in pipeline files
Use variables for reusable values — Bucket names, project IDs, file paths

Pipeline v2 Features

Pipeline v2 extends the basic format with advanced flow control. All v2 fields are optional and backward compatible with the v1 format described above.

Retry Policy

Automatically retry a step on failure with configurable delay:

steps:
  - name: Upload model
    command: object upload my-bucket model.rvt
    retry:
      max_attempts: 3
      delay: 5s

Step Timeout

Set a maximum execution time for a step:

steps:
  - name: Translate model
    command: translate start $URN --wait
    timeout: 30m

Conditional Execution

Run or skip steps based on expressions that reference previous step results:

steps:
  - name: Notify on success
    command: api post /webhooks/notify
    if: "${{ steps.translate.exit_code == 0 }}"

  - name: Cleanup on failure
    command: object delete my-bucket model.rvt
    unless: "${{ steps.translate.exit_code == 0 }}"

Parallel Steps

Execute multiple steps concurrently:

steps:
  - name: Upload files in parallel
    parallel:
      - name: Upload model A
        command: object upload my-bucket model-a.rvt
      - name: Upload model B
        command: object upload my-bucket model-b.rvt

For-Each Loops

Iterate over a list of items with optional concurrency control:

steps:
  - name: Process models
    for_each:
      items: ["model-a.rvt", "model-b.rvt", "model-c.rvt"]
      variable: MODEL
      max_concurrency: 2
      template:
        name: "Upload ${MODEL}"
        command: "object upload my-bucket ${MODEL}"

Expressions

Pipeline expressions use ${{ }} syntax and can reference step results:

${{ steps.<id>.exit_code }} — exit code of a previous step
${{ env.VARIABLE }} — environment variable
Supports comparison operators: ==, !=, >, <, >=, <=

Next Steps

Examples — More workflow examples
Exit Codes — CI/CD integration