Getting Started#
To get started developing a tsdat pipeline, we suggest following the steps below.
- Get a template
- Configure the template
- Use the template
Choose a template#
To create a standalone pipeline repository from the installed tsdat package:
pip install tsdat
tsdat init my-pipelines
cd my-pipelines
pip install -r requirements-dev.txt
tsdat create-pipeline ingest # or: tsdat create-pipeline vap
tsdat init creates a new project directory with a working example and
pipeline templates. Run pytest to check the project; see its README for an example
ingest command. Run tsdat create-pipeline from the project root to add pipelines.
Alternatively, you can use a GitHub repository template. You can find a list of template repositories for tsdat here.
The most used and most flexible template is the Pipeline Template repository, so we recommend starting there.
Once you have selected the template to use, select the "Use this template" button to create a new repository at your specified location with the template contents.
Note
You must have a github account and be signed-in to see this button. You can still click to download the code as a zip file and follow the rest of the setup instructions, but you will be lacking certain features (e.g., automated testing via github actions)

Configure the Template#
If you created the repository through GitHub, clone it to your local desktop to start developing. Locally initialized repositories are ready to use after installing dependencies. By default, the repository template comes pre-configured to run out-of-the-box on an example dataset.
We recommend that reading through the template's README and examining the included example before starting to configure the template. The template README will include end-to-end instructions on how to customize the template and create your own pipelines, but for those who learn best doing things hands-on, we recommend starting with the pipeline template tutorial for a complete walkthrough of setting up and configuring the pipeline-template repository.
See the Configuring Tsdat section for more information on tsdat configuration files and code customizations.
Using the template#
Once tsdat is installed and your template is configured, run tsdat ingest or tsdat vap
from the pipeline repository root to process data. To see the available commands, run:
tsdat ingest FILE... runs each matching pipeline once per file by default. Add
--clump to run each pipeline once with all files matching its triggers; add
--multidispatch to allow the same file to go to multiple matching pipelines.
Unmatched files are skipped. tsdat vap CONFIG --begin DATE --end DATE runs a
transformation pipeline for a date interval.
For detailed examples of how to set up and use tsdat see pipeline customization and our other tutorials.