Scaling up computation for research workflows across platforms

As research software engineers, we face a fundamental challenge: building scalable, maintainable computational pipelines that can grow from prototype to production while remaining reproducible and efficient. Traditional approaches—shell scripts with hardcoded paths, manual job submission, ad-hoc resource allocation—quickly become technical debt that impedes both research velocity and software quality. Workflow Managment Software like Nextflow offers a software engineering solution to this problem, providing declarative workflow definitions, separation of concerns between logic and execution, and infrastructure-as-code principles for local, cloud and HPC environments. This presentation examines Nextflow resource management on SLURM clusters through an engineering lens, focusing on design patterns, configuration management, and system optimization. Using make_lastz_chains—a computationally demanding genome alignment pipeline—we'll explore how proper executor configuration, process-level resource directives, and monitoring strategies create robust, scalable research software. We'll cover architectural considerations for pipeline design, common anti-patterns that lead to resource contention or underutilization, and engineering best practices for deploying workflows across HPC infrastructures. The goal is equipping you with software engineering principles that transform research code into reliable, scalable research software.

Join us on Wednesday, Mar. 4 at 1pm MST on Zoom!