CI/CD for Data Engineering: A Hands-On GitHub Actions Guide

Most data teams version their code in Git, then deploy it by hand: someone SSHs into a server, pulls the latest branch, and hopes nothing breaks. That works until the day a broken SQL model or a half-deployed script quietly corrupts the numbers your business runs on.

This guide is for data engineers and data leads who want to fix that. I'll explain why CI/CD is mandatory for any data project, what a data CI pipeline should actually check, and give you a working template that provisions an EC2 instance with Terraform and deploys to it with GitHub Actions on every push.

What is CI/CD in data engineering?#

CI/CD stands for continuous integration and continuous delivery (or deployment). In software, it means every commit is built, tested and shipped through an automated pipeline.

In data engineering, the "code" is wider than an application. It includes Python ingestion scripts, Airflow DAGs, dbt models, AWS Glue jobs, Terraform for your warehouse and buckets, and the configuration that ties them together. CI/CD applies the same discipline to all of it.

The key idea: nobody deploys from their laptop. Git is the single source of truth, and the pipeline is the only thing allowed to touch production.

Why is CI/CD mandatory for data projects?#

I say mandatory, not "nice to have". Here is why, in five points.

1. Faster, smaller releases#

As soon as code is merged, the pipeline tests and deploys it. Fixes and new models reach production in minutes instead of waiting for someone to find time for a manual deploy.

Smaller, frequent releases are also easier to debug. If a dashboard breaks after a deploy that changed one model, you know where to look.

2. Consistent, repeatable deployments#

A pipeline runs the same steps in the same order every time. No forgotten pip install, no file copied to the wrong folder, no "it worked on my machine".

That consistency is what makes releases predictable, which matters a lot when finance or marketing reads your tables every morning.

3. Automated testing before anything hits production#

CI is where you catch bugs before they reach your users. For data, that means more than unit tests: it means checking that SQL compiles, that models still build, and that data tests (not_null, unique, accepted values) still pass.

A bad deploy in a web app shows an error page. A bad deploy in a data pipeline often shows nothing at all, just wrong numbers. That is exactly why automated checks matter more in data, not less.

4. Easy rollback#

Because every deploy maps to a Git commit, rolling back is a git revert and a new pipeline run. You return to a known-good state in minutes, without anyone trying to remember what changed on the server.

5. Collaboration and visibility#

Every change goes through a pull request, gets reviewed, and leaves a trace. The pipeline shows who deployed what and when. This is also what lets a team grow beyond one person who "knows how to deploy".

It is also the foundation for anything AI-related. An AI agent or analytics assistant is only as reliable as the tables it reads, and tested, versioned pipelines are how you keep those tables trustworthy. That is a big part of what we build in our data foundations work.

What should a data CI/CD pipeline check?#

Here is the checklist I use. You do not need all of it on day one, but each row removes a class of production incidents.

Tutorial: deploy code to EC2 with Terraform and GitHub Actions#

Now the hands-on part. This is a deliberately simple proof of concept: Terraform creates an EC2 instance, and a GitHub Actions workflow copies your repository to that instance on every push to main.

It is the same pattern you would use to deploy ingestion scripts or a small scheduler to a server. The full code is on GitHub: Dorianteffo/deploy_ec2. I've updated the snippets below for current versions.

Project structure#

deploy_ec2/
├── .github/
│   └── workflows/
│       └── deploy_ec2.yml
├── terraform/
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
├── Makefile
└── test.sh

Step 1: Create the EC2 instance with Terraform#

First, the AWS resources. Could you click through the console? Sure. Should you? No. Everything should be code and version-controlled, so we use Terraform.

To create an EC2 instance, you need three things: an AMI, a key pair for SSH, and a security group. Here is main.tf:

terraform {
  required_version = ">= 1.10"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0"
    }
  }

  # Remote state in S3. use_lockfile enables native S3 locking (Terraform 1.10+), no DynamoDB table needed.
  backend "s3" {
    bucket       = "your-tf-state-bucket"
    key          = "deploy-ec2/terraform.tfstate"
    region       = "eu-west-3"
    encrypt      = true
    use_lockfile = true
  }
}

provider "aws" {
  region = "eu-west-3"
}

# Latest Ubuntu 24.04 LTS AMI published by Canonical
data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["099720109477"] # Canonical

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

# Generate an SSH key pair. The private key ends up in Terraform state, which is why the state bucket is encrypted.
resource "tls_private_key" "deploy" {
  algorithm = "ED25519"
}

resource "aws_key_pair" "deploy" {
  key_name   = var.key_name
  public_key = tls_private_key.deploy.public_key_openssh
}

resource "aws_security_group" "deploy" {
  name        = "deploy-ec2-sg"
  description = "Allow SSH for deployments"

  ingress {
    description = "SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = var.ssh_allowed_cidrs
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "deploy-ec2-sg"
  }
}

resource "aws_instance" "app" {
  ami                    = data.aws_ami.ubuntu.id
  instance_type          = var.instance_type
  key_name               = aws_key_pair.deploy.key_name
  vpc_security_group_ids = [aws_security_group.deploy.id]

  tags = {
    Name = "deploy-ec2"
  }
}

Don't be scared by the length. About a third of it is copied straight from the Terraform AWS provider documentation.

The real skill is knowing what a resource needs. Once you know that an EC2 instance needs an AMI, a key pair and a security group, the docs give you the rest.

Then variables.tf:

variable "instance_type" {
  type        = string
  default     = "t3.micro"
  description = "EC2 instance type"
}

variable "key_name" {
  type        = string
  default     = "deploy-key"
  description = "Name of the EC2 key pair"
}

variable "ssh_allowed_cidrs" {
  type        = list(string)
  default     = ["0.0.0.0/0"] # POC only. See the SSM section below for how to close port 22.
  description = "CIDR blocks allowed to reach port 22"
}

Step 2: Output the values GitHub Actions needs#

The deploy workflow needs three values:

• REMOTE_HOST: the public DNS of the instance.
• REMOTE_USER: ubuntu, because we use the Ubuntu AMI.
• SERVER_SSH_KEY: the private key matching the instance's key pair. Terraform can output the first and the last one. Here is outputs.tf:

output "ec2_public_dns" {
  description = "Public DNS of the EC2 instance"
  value       = aws_instance.app.public_dns
}

output "private_key" {
  description = "Private SSH key for the deploy user"
  value       = tls_private_key.deploy.private_key_openssh
  sensitive   = true
}

Step 3: Wrap the Terraform commands in a Makefile#

I like a small Makefile so nobody has to remember flags:

tf-init:
	terraform -chdir=./terraform init

tf-plan:
	terraform -chdir=./terraform plan

tf-apply:
	terraform -chdir=./terraform apply

ec2-private-key:
	terraform -chdir=./terraform output -raw private_key

ec2-dns:
	terraform -chdir=./terraform output -raw ec2_public_dns

tf-destroy:
	terraform -chdir=./terraform destroy

Create the infrastructure:

make tf-init
make tf-plan
make tf-apply

Then read the two values you need:

make ec2-private-key   # copy the full output, including the BEGIN and END lines
make ec2-dns           # copy the public DNS

Step 4: Store the values as GitHub secrets#

In your repository, go to Settings, then Secrets and variables, then Actions, and create three repository secrets:

• SERVER_SSH_KEY: the output of make ec2-private-key.
• REMOTE_HOST: the output of make ec2-dns.
• REMOTE_USER: ubuntu. Or do it from the terminal with the GitHub CLI:

terraform -chdir=./terraform output -raw private_key | gh secret set SERVER_SSH_KEY
terraform -chdir=./terraform output -raw ec2_public_dns | gh secret set REMOTE_HOST
gh secret set REMOTE_USER --body "ubuntu"

Step 5: Write the GitHub Actions deploy workflow#

Now the CD part. Create .github/workflows/deploy_ec2.yml:

name: deploy-to-ec2

on:
  push:
    branches: [main]
  workflow_dispatch:

concurrency:
  group: deploy-ec2
  cancel-in-progress: false

permissions:
  contents: read

jobs:
  deploy:
    name: Deploy to EC2
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v5

      - name: Deploy to server
        uses: easingthemes/ssh-deploy@v5
        with:
          SSH_PRIVATE_KEY: ${{ secrets.SERVER_SSH_KEY }}
          REMOTE_HOST: ${{ secrets.REMOTE_HOST }}
          REMOTE_USER: ${{ secrets.REMOTE_USER }}
          SOURCE: "./"
          TARGET: "/home/ubuntu/app"
          EXCLUDE: "/.git/, /terraform/"
          SCRIPT_AFTER: |
            bash /home/ubuntu/app/test.sh

A few details that matter:

• Pin actions to a version (@v5), never @main. A moving branch can change under you.
• concurrency stops two deploys from writing to the server at the same time.
• permissions: contents: read gives the workflow token the minimum it needs.
• SCRIPT_AFTER runs a command on the server once files are copied, which is where you would restart a service or reinstall dependencies. For more options, read the easingthemes/ssh-deploy page on the GitHub Marketplace.

Step 6: Push and watch it deploy#

To test the pipeline, add a tiny script at the root of the repo, test.sh:

#!/bin/bash
echo "Deployed successfully on $(hostname) at $(date -u)"

Then commit and push:

git add .
git commit -m "Test EC2 deployment"
git push origin main

Open the Actions tab in GitHub. You should see the deploy-to-ec2 workflow run, copy the files to /home/ubuntu/app and print the success line from test.sh in the logs. From now on, every push to main deploys automatically.

How do you add CI checks before deploying?#

The template above is pure CD: it deploys whatever lands on main. In a real data project, you want CI on every pull request so that broken code never reaches main in the first place.

Here is a CI workflow for a typical Python plus dbt plus Terraform repository:

name: ci

on:
  pull_request:
    branches: [main]

permissions:
  contents: read

jobs:
  python:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: astral-sh/setup-uv@v6
      - name: Lint and format check
        run: |
          uvx ruff check .
          uvx ruff format --check .
      - name: Unit tests
        run: uv run pytest -q

  terraform:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: hashicorp/setup-terraform@v3
      - name: Format and validate
        run: |
          terraform -chdir=./terraform fmt -check -recursive
          terraform -chdir=./terraform init -backend=false
          terraform -chdir=./terraform validate

  dbt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install dbt
        run: pip install dbt-core dbt-bigquery
      - name: Build only modified models (Slim CI)
        run: |
          dbt deps
          dbt build --select state:modified+ --defer --state ./prod-artifacts --target ci

The dbt job is the one that pays off most. With state:modified+, dbt only builds the models you changed plus everything downstream of them, and --defer reads unchanged parents from production. You need the production manifest.json in ./prod-artifacts (download it from S3 or GCS in an earlier step, or let dbt Cloud handle it) and a ci target in profiles.yml that writes to a separate CI schema. CI stays fast even with hundreds of models.

Then, in GitHub, turn on branch protection for main and require these checks to pass before merging. That single setting is what turns CI from a suggestion into a rule.

If your pipelines run serverless, the same approach works for Lambda. I cover it step by step in CI/CD for AWS Lambda with GitHub Actions.

Should you still deploy over SSH in 2026?#

The SSH template is great for learning, but it has two weak spots.

First, GitHub-hosted runners don't have fixed IP addresses, so port 22 has to stay open to a very wide range (in the POC, the whole internet). Second, the private key lives in Terraform state and in GitHub secrets for as long as the instance exists.

The modern setup removes both:

• OIDC instead of stored credentials. GitHub Actions requests a short-lived token from AWS for each run. No AWS access keys in secrets, ever.
• AWS Systems Manager (SSM) instead of SSH. The workflow uploads the build to S3, then asks SSM to run the deploy commands on the instance. Port 22 stays closed. Official Ubuntu AMIs ship with the SSM agent; the instance only needs an IAM role with the AmazonSSMManagedInstanceCore policy. The Terraform for the GitHub OIDC trust looks like this:

resource "aws_iam_openid_connect_provider" "github" {
  url            = "https://token.actions.githubusercontent.com"
  client_id_list = ["sts.amazonaws.com"]
}

data "aws_iam_policy_document" "github_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    # Only the main branch of this repository can assume the role
    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:your-org/your-repo:ref:refs/heads/main"]
    }
  }
}

resource "aws_iam_role" "github_deploy" {
  name               = "github-deploy"
  assume_role_policy = data.aws_iam_policy_document.github_trust.json
}

Attach a policy to github_deploy that allows s3:PutObject on your artifact bucket and ssm:SendCommand on the instance. Then the deploy job becomes:

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write   # required for OIDC
      contents: read
    steps:
      - uses: actions/checkout@v5

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-deploy
          aws-region: eu-west-3

      - name: Upload build to S3
        run: |
          tar -czf app.tar.gz --exclude=.git --exclude=terraform .
          aws s3 cp app.tar.gz s3://your-artifact-bucket/app/${{ github.sha }}.tar.gz

      - name: Deploy with SSM
        run: |
          SHA=${{ github.sha }}
          aws ssm send-command \
            --document-name "AWS-RunShellScript" \
            --targets "Key=tag:Name,Values=deploy-ec2" \
            --comment "Deploy $SHA" \
            --parameters commands="aws s3 cp s3://your-artifact-bucket/app/$SHA.tar.gz /tmp/app.tar.gz && mkdir -p /home/ubuntu/app && tar -xzf /tmp/app.tar.gz -C /home/ubuntu/app && bash /home/ubuntu/app/test.sh"

The instance role also needs s3:GetObject on the artifact bucket so it can pull the build, and the instance needs the AWS CLI (install it once in user data with snap install aws-cli --classic).

Here is how the options compare:

Honestly, for most data teams the end state is not a long-lived EC2 instance at all. It is containers or managed services, with the same CI/CD principles on top. The pattern you learned here carries over unchanged. If you are deciding what that end state looks like, I wrote about it in how to build a modern data infrastructure.

FAQ#

What is the difference between CI and CD in data engineering?#

CI (continuous integration) runs on every pull request and checks that the change is safe: linting, unit tests, a dbt build of modified models, terraform plan. CD (continuous delivery or deployment) runs after merge and ships the change: terraform apply, deploying code, syncing DAGs. CI protects main, CD keeps production in sync with main.

How do you test data pipelines in CI?#

Use three layers. Unit tests (pytest) cover Python transformation logic with small fixtures. dbt tests (not_null, unique, relationships, custom tests) validate the data your models produce, ideally on a CI schema with state:modified+. Integration checks, such as an Airflow DAG import test, confirm everything loads and wires together.

Is GitHub Actions good enough for data engineering CI/CD?#

Yes, for most teams. It handles Python, dbt, Terraform and AWS deploys well, supports OIDC to AWS, GCP and Azure, and lives next to your code. GitLab CI works just as well if you are on GitLab. You only need a dedicated orchestrator for running the pipelines themselves, not for deploying them.

Should I store AWS access keys in GitHub secrets?#

No. Use GitHub's OIDC integration with an IAM role that only your repository and branch can assume. Each workflow run gets temporary credentials that expire on their own, so there is nothing long-lived to leak or rotate.

Do small data teams really need CI/CD?#

Especially small teams. When one or two people run the whole data stack, there is no one to catch a bad deploy by eye. A basic pipeline (lint, tests, dbt build, automated deploy) takes a day to set up and saves you from silent breakages for years.

Key takeaways#

• Treat every part of a data project as code: pipelines, SQL models, DAGs and infrastructure.
• CI on pull requests catches broken code and broken data before production. CD on merge removes manual deploys.
• Terraform plus GitHub Actions gets you from zero to automated deploys on AWS in an afternoon.
• Move from SSH keys to OIDC and SSM (or containers) once the POC works.
• Protect main with required checks. That is what makes the whole system stick. CI/CD is not the exciting part of a data stack, but it is the part that keeps every dashboard and model trustworthy. If you want a production-ready setup (Terraform, GitHub Actions, dbt Slim CI, OIDC) built for your team and handed over with docs, book a call with Dublo. We also run it end to end as part of our data integration service.