Managing multiple projects with dg

info

dg and Dagster Components are under active development. You may encounter feature gaps, and the APIs may change. To report issues or give feedback, please join the #dg-components channel in the Dagster Community Slack.

note

If you're just getting started, we recommend scaffolding a single project instead of a workspace with multiple projects.

If you need to collaborate with multiple teams, or work with conflicting dependencies that require isolation from each other, you can scaffold a workspace directory that contains multiple projects, each with their own separate Python environment.

A workspace directory contains a root dg.toml with workspace-level settings, and a projects directory with one or more projects.

note

A workspace does not define a Python environment by default. Instead, Python environments are defined per project.

Scaffold a new workspace and first project

To scaffold a new workspace called dagster-workspace, run uvx create-dagster workspace:

uvx create-dagster workspace dagster-workspace && cd dagster-workspace

Creating a Dagster workspace at dagster-workspace.
Scaffolded files for Dagster workspace at /.../dagster-workspace.

Now we'll create a project inside our workspace called project-1. Run uvx create-dagster project with the --python-environment uv_managed option. You will be prompted for the name of the project:

uvx create-dagster project --python-environment uv_managed projects/project-1

Creating a Dagster project at /.../dagster-workspace/projects/project-1.
Scaffolded files for Dagster project at /.../dagster-workspace/projects/project-1.
...

note

Currently dg workspaces only support projects using uv with project.python_environment.uv_managed = true. This means that the Python environment for the workspace is managed by uv, and subprocesses are launched by uv run, ignoring the activated virtual environment. If all projects in a workspace do not conform to this, you will likely encounter errors.

This will create a new directory called dagster-workspace with a projects subdirectory that contains project-1. It will also set up a new uv-managed Python environment for this project.

Review workspace structure

The new workspace has the following structure:

tree

.
├── dg.toml
└── projects
    └── project-1
        ├── pyproject.toml
        ├── src
        │   └── project_1
        │       ├── __init__.py
        │       ├── components
        │       │   └── __init__.py
        │       └── defs
        │           └── __init__.py
        ├── tests
        │   └── __init__.py
        └── uv.lock

...

The dg.toml file for the dagster-workspace folder contains a directory_type = "workspace" setting that marks this directory as a workspace:

dagster-workspace/dg.toml

directory_type = "workspace"

[workspace]


[[workspace.projects]]
path = "projects/project-1"

note

project-1 also contains a virtual environment directory called .venv that is not shown above. This environment is managed by uv and its contents are specified in the uv.lock file.

The project-1 directory contains a pyproject.toml file with a tool.dg.directory_type = "project" section that defines it as a dg project:

dagster-workspace/projects/project-1/pyproject.toml

...
[tool.dg]
directory_type = "project"

[tool.dg.project]
root_module = "project_1"
...

Add a second project to the workspace

As noted above, environments are scoped per project. dg commands will only use the environment of project-1 when you are inside the project-1 directory.

Let's create another project:

uvx create-dagster project projects/project-2 --python-environment uv_managed

Creating a Dagster project at /.../dagster-workspace/projects/project-2.
Scaffolded files for Dagster project at /.../dagster-workspace/projects/project-2.
...

Now we have two projects. We can list them with:

dg list project

projects/project-1
projects/project-2

Load workspace with dg

Finally, let's load up our two projects with dg dev. When we run dg dev from the workspace root, it will automatically recognize the projects in your workspace and launch them in their respective environments. However, since the workspace root does not have an associated Python environment, we'll need to use the dg executable from one of our project environments. We'll use project-1. Let's activate that virtual environment and then launch dg dev:

source projects/project-1/.venv/bin/activate && dg dev

note

More streamlined python environment management at the workspace level is under development.