CI/CD Pipelines

CI, CD, and CD — Three Ideas

A pipeline is an automated sequence of steps that takes a commit and moves it toward production. The acronym CI/CD bundles three related but distinct practices that people constantly confuse.

Pipeline Anatomy

Most tools model a pipeline as stages that run in order, each containing jobs that run in parallel. A job is a set of shell steps executed on a runner. The example below uses GitLab CI (.gitlab-ci.yml) syntax, which maps cleanly onto other tools.

stages:
  - build
  - test
  - deploy

variables:
  NODE_VERSION: "22"

build:
  stage: build
  image: node:22-alpine
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

unit-test:
  stage: test
  image: node:22-alpine
  script:
    - npm ci
    - npm test -- --coverage
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'

deploy-staging:
  stage: deploy
  script:
    - ./deploy.sh staging
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - main

Stages: Build, Test, Deploy

The build stage compiles code and produces artifacts. The test stage runs unit, integration, lint, and security scans — often in parallel jobs. The deploy stage ships the artifact to an environment. A failing job in an early stage short-circuits the pipeline so later stages never run.

Triggers

Pipelines are started by triggers. Choosing the right trigger keeps feedback fast without wasting runner minutes.

# Run only on merge requests targeting main, plus tags
test:
  stage: test
  script: npm test
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_TAG'
    - if: '$CI_COMMIT_BRANCH == "main"'

Caching and Artifacts

These two look similar but serve opposite purposes. Cache speeds up future pipelines by reusing dependencies; artifacts pass build output to later stages of the same pipeline.

build:
  stage: build
  cache:
    key:
      files:
        - package-lock.json    # cache key busts when lockfile changes
    paths:
      - .npm/
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run build
  artifacts:
    paths:
      - dist/

Environments and Promotion

An environment is a named deployment target (dev, staging, production). A single artifact is promoted through environments — you build once and deploy the same immutable artifact everywhere, changing only configuration. Rebuilding per environment is an anti-pattern because it can introduce drift.

Secrets Management

Never commit credentials. Store them as masked, protected CI variables or pull them at runtime from a secrets manager (Vault, AWS Secrets Manager, cloud OIDC). Prefer short-lived OIDC tokens over long-lived static keys.

deploy-prod:
  stage: deploy
  id_tokens:
    AWS_ID_TOKEN:               # OIDC token, no static keys stored
      aud: https://sts.amazonaws.com
  script:
    - aws sts assume-role-with-web-identity \
        --role-arn "$AWS_ROLE_ARN" \
        --web-identity-token "$AWS_ID_TOKEN" \
        --role-session-name ci
    - ./deploy.sh production

Matrix Builds

A matrix fans a single job into many variants — testing across language versions, OSes, or configurations in parallel.

test:
  stage: test
  image: node:$NODE_VERSION
  parallel:
    matrix:
      - NODE_VERSION: ["20", "22", "24"]
        DB: ["postgres:16", "postgres:17"]
  services:
    - $DB
  script:
    - npm ci
    - npm test

Gates and Approvals

A manual gate pauses the pipeline until an authorized person approves. Common on production deploys in a continuous delivery setup. Gates can also be automated quality checks (coverage thresholds, security scan severity, DAST results).

deploy-prod:
  stage: deploy
  script: ./deploy.sh production
  environment:
    name: production
  when: manual          # requires a human click
  allow_failure: false  # blocks the pipeline until approved
  rules:
    - if: '$CI_COMMIT_TAG'

Deployment Strategies

How you cut over from old to new version determines your blast radius and rollback speed.

# Canary: shift traffic in steps, promote only if metrics stay healthy
deploy-canary:
  stage: deploy
  script:
    - ./deploy.sh --strategy canary --weight 10
    - ./check-slo.sh --error-rate-max 1% --window 5m
    - ./deploy.sh --strategy canary --weight 50
    - ./check-slo.sh --error-rate-max 1% --window 5m
    - ./deploy.sh --strategy canary --weight 100  # full promotion

Practice Exercises

1. Write a three-stage pipeline (build, test, deploy) that only deploys when the pipeline runs on the main branch.
2. Add dependency caching keyed on your lockfile and confirm a second run is faster.
3. Convert your test job into a matrix that runs against three language versions in parallel.
4. Add a manual approval gate before the production deploy job, restricted to tag pipelines.
5. Replace static cloud credentials with an OIDC-based short-lived token in your deploy job.
6. Sketch a canary rollout that shifts traffic 10 → 50 → 100% with an automated SLO check between steps.