Branch Deployments automatically create staging environments of your Dagster code, right in Dagster Cloud. For every push to a branch in your git repository, Dagster Cloud will create a unique deployment, allowing you to preview the changes in the branch in real-time.
Think of a branch deployment as a branch of your data platform, one where you can preview changes without impacting production or overwriting a testing environment.
Let's take a closer look:
In your git repository, a new branch is created off of main
. In the example above, this branch is named feature-1
.
Dagster Cloud is notified of the push and creates a branch deployment named feature-1
. The branch deployment functions just like your production
deployment of Dagster Cloud, but contains the Dagster code changes from the feature-1
branch.
In this example, the feature-1
branch deployment 'talks' to a cloned schema
in a database. This is completely separate from the prod schema
associated with the production
deployment.
For every push to the feature-1
branch, the feature-1
branch deployment in Dagster Cloud is re-built and re-deployed.
Now that you know how Branch Deployments work, why should you use them?
To use Branch Deployments, you'll need a Dagster Cloud account.
Branch Deployments can be used with any git or CI provider. However, setup is easiest with the Dagster GitHub app as parts of the process are automated. Refer to the Setting up Branch Deployments section for more info.
The following aren't currently supported for Branch Deployments:
Output created from a branch deployment - such a database, table, etc. - won't be automatically removed from storage once a branch is merged or closed. Refer to the Best practices section for info on how to handle this.
There are currently two ways to set up Branch Deployoments for Dagster Cloud. In the table below:
Platform | How it works | May be a good fit if... |
---|---|---|
GitHub | GitHub Actions |
|
Other git/CI platform | dagster-cloud CLI |
|
Once configured, branch deployments can be accessed:
Every pull request in the repository contains a View in Cloud link:
Clicking the link will open a branch deployment - or a preview of the changes - in Dagster Cloud.
You can also access branch deployments directly in Dagster Cloud from the deployment switcher:
To ensure the best experience when using Branch Deployments, we recommend:
Configuring jobs based on environment. Dagster automatically sets environment variables containing deployment metadata, allowing you to parameterize jobs based on the executing environment. Use these variables in your jobs to configure things like connection credentials, database, and so on. This practice will allow you to use Branch Deployments without impacting production data.
Creating jobs to automate output cleanup. As Branch Deployments don't automatically remove the output they create, you may want to create an additional Dagster job to perform the cleanup.
Want some help with implementation? Check out the Testing against production with Dagster Cloud Branch Deployments guide for a step-by-step look at implementing these best practices in your own data pipelines.