Snakemake Primer

Scientific results need to be reproducible, but this can be challenging in computational biology where software versions and system configurations vary. Our workflow is built using a combination of tools that solve this problem.

Snakemake: A Cookbook for Bioinformatics Workflows

Image Credit: https://commons.wikimedia.org/wiki/File:Snakemake_logo_dark.png

At its core, our workflow is managed by Snakemake. Think of a Snakemake workflow (defined in a file called a Snakefile) as a detailed recipe for a complex scientific analysis. Each step in the analysis, like aligning reads or counting genes, is a “rule” in the recipe. Each rule clearly defines its inputs (the ingredients) and the outputs it will produce (a prepared component of the final dish).

Image Credit: https://www.rc.virginia.edu/userinfo/hpc/software/snakemake/

Snakemake is the master chef that reads this recipe. It understands the dependencies between all the steps and uses the Snakefile to intelligently automate the entire analysis, providing several key advantages:

• Automation: It automatically determines the correct order to run each rule. This prevents human error and saves you from manually executing dozens of commands.
• Efficiency: It can run multiple independent rules in parallel, dramatically speeding up the analysis, especially on multi-core computers or HPC clusters.
• Reproducibility: If an input file or a parameter is changed, Snakemake knows exactly which steps need to be re-run, ensuring the results are always up-to-date.

Before executing anything, Snakemake first examines the entire workflow, noting all the inputs and outputs of every rule. It uses this information to build a map of job dependencies called a Directed Acyclic Graph (DAG). This DAG is the master plan that determines exactly what steps need to be run and in what order.

Chicken Tikka Masala as a DAG

A recipe has a list of ingredients (inputs) and steps that must be followed in order to produce a dish (output). This is an example of a directed acyclic graph (DAG) for a chicken tikka masala recipe.

Image Credit: https://fluidvisual.com/flowchartrecipes/

The tucca-rna-seq Workflow DAG

Snakemake defines each step as a “rule” with defined inputs and outputs, creating a directed acyclic graph (DAG) from your initial raw reads to the workflow’s various outputs.

This image shows the workflow’s rulegraph, which illustrates the dependencies between rules, providing a high-level blueprint of the analysis. A full DAG of jobs, by contrast, would be much larger, showing a specific job for each individual input file.

Snakemake Links Docs and API

The workflow management system that orchestrates the analysis, using Conda to create reproducible environments for each step.

This is the most important page to visit when learning how to execute, debug, and configure your workflow.

Command Line Interface (CLI) DocsOfficial Documentation Home

For more advanced needs, or when information isn’t in the main docs, the API reference is useful.

API Reference

Learn Snakemake (Recommended Tutorials)

While the official tutorial is a great start, we recommend the one from The Carpentries for its detailed, bioinformatics-focused approach.

Offical Beginner's TutorialSnakemake Bioinformatics Tutorial (The Carpentries)

The Reproducibility Problem: Why Your “Oven” Matters in Bioinformatics

Imagine two people trying to bake the exact same cake using the same recipe. One person uses a modern convection oven, while the other uses an older gas oven. Even with the same ingredients, the different ovens (their “environment”) can produce slightly different cakes.

This is a common problem in bioinformatics. A script that works perfectly on one computer might fail or produce different results on another. This is because the “environment”—including the operating system and the specific versions of software tools—is different.

The “It Worked on My Machine” Problem

In this example, Person #1 can run the analysis successfully. When Person #2 tries to run the same analysis on their computer, it fails because of different software versions. This is a classic reproducibility issue.

Conda: The Universal Shopping List

To begin to solve this, we can use a package manager like Conda.

If the Snakefile is our recipe, think of Conda as a universal, automated shopping list. For each step in our recipe, Conda reads a list of required software “ingredients” (e.g., a specific version of a read aligner or a quality control tool) from a dedicated environment.yaml file.

It then automatically:

1. Finds the exact versions of the software specified.
2. Installs them into an isolated, self-contained “environment”.
3. Ensures that the software for one step doesn’t conflict with the software for another.

This way, when Person #1 shares their Snakefile and their Conda environment files, Person #2 can recreate the exact same software setup, making the analysis much more likely to run successfully.

However, Conda only manages the software inside a given operating system. What if the two computers have fundamentally different operating systems (e.g., Windows vs. macOS vs. Linux)? Tiny differences in system-level libraries can still cause problems. This brings us to the ultimate solution: containers.

Containers: The Perfect Kitchen

To solve the environment problem, we use containers.

A container is like a standardized kitchen. It doesn’t just contain the ingredients (the code and data), but also the exact kitchen appliances (specific software versions) and the recipe book (the operating system) needed to make the meal perfectly every time.

When you run the workflow using a container, you’re running it in a consistent, pre-configured environment. This ensures that the analysis will produce the exact same results, regardless of what machine it’s run on. This is the gold standard for reproducible science.

How Containers Create Consistency

On the left, traditional applications rely on the host operating system’s libraries, which can vary. On the right, containers bundle their own libraries, ensuring the application always runs in the exact same environment.

Helpful Video Resources

Reproducible Data Analysis with Snakemake

In this video, Johannes Köster, the creator of Snakemake, provides a comprehensive tutorial on how to use the workflow management system for reproducible data analysis. He demonstrates how to define rules, manage dependencies, and use Conda environments to ensure that a workflow is portable and scalable.

An introduction to snakemake: a tool for automating and streamlining your analyses

This video and GitHub repository provide a beginner-friendly introduction to Snakemake, a powerful tool for automating and streamlining data analyses. The content is designed to be accessible, offering a sample workflow and clear instructions to help users learn how to manage their computational biology projects more efficiently.

Git Repo

Snakemake Tips & Tricks (Playlist)

This playlist, “Snakemake Tips & Tricks,” created by “actually scientist,” offers a series of tutorials on using Snakemake for computational workflows. The videos cover fundamental concepts like writing Snakefiles, understanding how Snakemake locates them, and using configuration files, as well as advanced topics such as incorporating external Snakefiles for modularization, employing wildcards for generalized workflows, utilizing multiple threads, managing resource usage, and submitting jobs to Slurm HPC clusters.

Full "Snakemake Tips & Tricks" Playlist