Repository navigation
Releases
v1.14 - Brass Chicken 🐔
Template
- Fixed an issue regarding explicit disabling of unused container engines [#972]
- Removed trailing slash from
params.igenomes_baseto yield valid s3 paths (previous paths work with Nextflow but not aws cli) - Added a timestamp to the trace + timetime + report + dag filenames to fix overwrite issue on AWS
- Rewrite the
params_summary_log()function to properly ignore unset params and have nicer formatting [#971] - Fix overly strict
--max_timeformatting regex in template schema [#973] - Convert
dtodayin thecleanParametersfunction to make Duration objects like2dpass the validation [#858] - Added nextflow version to quick start section and adjusted
nf-core bump-version[#1032] - Use latest stable Nextflow version
21.04.0for CI tests instead of the-edgerelease
Download
- Fix bug in
nf-core downloadwhere image names were getting a hyphen innf-corewhich was breaking things. - Extensive new interactive prompts for all command line flags [#1027]
- It is now recommended to run
nf-core downloadwithout any cli options and follow prompts (though flags can be used to run non-interactively if you wish)
- It is now recommended to run
- New helper code to set
$NXF_SINGULARITY_CACHEDIRand add to.bashrcif desired [#1027]
Launch
- Strip values from
nf-core launchweb response which areFalseand have no default in the schema [#976] - Improve API caching code when polling the website, fixes noisy log message when waiting for a response [#1029]
- New interactive prompts for pipeline name [#1027]
Modules
- Added
tool_name_underscoreto the module template to allow TOOL_SUBTOOL inmain.nf[#1011] - Added
--conda-nameflag tonf-core modules createcommand to allow sidestepping questionary [#988] - Extended
nf-core modules lintfunctionality to check tags intest.ymland to look for a entry in thepytest_software.ymlfile - Update
modulescommands to use new test tag formattool/subtool - New modules lint test comparing the
functions.nffile to the template version - Modules installed from alternative sources are put in folders based on the name of the source repository
Linting
- Fix bug in nf-core lint config skipping for the
nextflow_configtest [#1019] - New
-k/--keycli option fornf-core lintto allow you to run only named lint tests, for faster local debugging - Merge markers lint test - ignore binary files, allow config to ignore specific files [#1040]
- New lint test to check if all defined pipeline parameters are mentioned in
main.nf[#1038] - Added fix to remove warnings about params that get converted from camelCase to camel-case [#1035]
- Added pipeline schema lint checks for missing parameter description and parameters outside of groups [#1017]
General
- Try to fix the fix for the automated sync when we submit too many PRs at once [#970]
- Rewrite how the tools documentation is deployed to the website, to allow multiple versions
- Created new Docker image for the tools cli package - see installation docs for details [#917]
- Ignore permission errors for setting up requests cache directories to allow starting with an invalid or read-only
HOMEdirectory
v1.13.3 - Copper Crocodile Resurrection 🐊 🚑 - [2021-03-24]
This is a patch release fixing some nasty bugs that crept into the code base for the nf-core helper tools. There are no changes to the nf-core pipeline template, so there should be no automated pipeline synchronisation pull-requests for this release.
New features
- Running tests twice with
nf-core modules create-test-ymlto catch unreproducible md5 sums [#890] nf-core modules createcommand - if no bioconda package is found, prompt user for a different bioconda package name- Updated module template
main.nfwith new test data paths
Bug fixes
- Fix sync error again where the Nextflow edge release needs to be used for some pipelines
- Fix bug with
nf-core lint --release(NameError: name 'os' is not defined) - Added linebreak to linting comment so that markdown header renders on PR comment properly
v1.13.2 - Copper Crocodile CPR 🐊 🤕
- Added better logging message if a user doesn't specificy the directory correctly with
nf-core modulescommands [#942] - Fixed parameter validation bug caused by JSONObject [#937]
- Fixed template creation error regarding file permissions [#932]
- Split the
create-lint-wftests up into separate steps in GitHub Actions to make the CI results easier to read - Added automated PR comments to the Markdown, YAML and Python lint CI tests to explain failures (tools and pipeline template)
- Make
nf-core lintsummary table borders coloured according to overall pass / fail status - Attempted a fix for the automated sync when we submit too many PRs at once [#911]
v1.13.1 - Copper Crocodile Patch 🐊 🏴☠️
Minor patch release to scoop up a small few errors discovered after the release.
- Fixed bug in pipeline linting markdown output that gets posted to PR comments [#914]
- Made text for the PR branch CI check less verbose with a TLDR in bold at the top
- A number of minor tweaks to the new
nf-core modules lintcode
v1.13 - Copper Crocodile
When we do releases, we sometimes like to go big. This is one of those times.
This is a truly huge release of nf-core/tools with a massive amount of new functionality. Please be sure to read the changelog and updated readme carefully!
Note to developers - please run and push the results of nf-core lint --fix before attempting to merge the automated synchronisation PR. This should reduce the size of the diff and make merging a lot easier.
The template and associated updates in this release is for Nextflow DSL1 pipelines. This is likely to be one of the last releases before we officially switch to DSL2. For those of you already with DSL2 pipelines you may need to treat the automated merge PR very carefully. Ask for help on Slack if in any doubt.
Template
- Major new feature - Validation of pipeline parameters [#426]
- The addition runs as soon as the pipeline launches and checks the pipeline input parameters two main things:
- No parameters are supplied that share a name with core Nextflow options (eg.
--resumeinstead of-resume) - Supplied parameters validate against the pipeline JSON schema (eg. correct variable types, required values)
- No parameters are supplied that share a name with core Nextflow options (eg.
- If either parameter validation fails or the pipeline has errors, a warning is given about any unexpected parameters found which are not described in the pipeline schema.
- This behaviour can be disabled by using
--validate_params false
- The addition runs as soon as the pipeline launches and checks the pipeline input parameters two main things:
- Added profiles to support the Charliecloud and Shifter container engines [#824]
- Note that Charliecloud requires Nextflow version
v21.03.0-edgeor later.
- Note that Charliecloud requires Nextflow version
- Profiles for container engines now explicitly disable all other engines [#867]
- Fixed typo in nf-core-lint CI that prevented the markdown summary from being automatically posted on PRs as a comment.
- Changed default for
--inputfromdata/*{1,2}.fastq.gztonull, as this is now validated by the schema as a required value. - Removed support for
--nameparameter for custom run names.- The same functionality for MultiQC still exists with the core Nextflow
-nameoption.
- The same functionality for MultiQC still exists with the core Nextflow
- Added to template docs about how to identify process name for resource customisation
- The parameters
--max_memoryand--max_timeare now validated against a regular expression [#793]- Must be written in the format
123.GB/456.hwith any of the prefixes listed in the Nextflow docs - Bare numbers no longer allowed, avoiding people from trying to specify GB and actually specifying bytes.
- Must be written in the format
- Switched from cookiecutter to Jinja2 [#880]
- Finally dropped the wonderful cookiecutter library that was behind the first pipeline template that led to nf-core [#880]
- Now rendering templates directly using Jinja, which is what cookiecutter was doing anyway
Modules
Initial addition of a number of new helper commands for working with DSL2 modules:
modules list- List available modulesmodules install- Install a module from nf-core/modulesmodules remove- Remove a module from a pipelinemodules create- Create a module from the templatemodules create-test-yml- Create thetest.ymlfile for a module with md5 sums, tags, commands and names addedmodules lint- Check a module against nf-core guidelines
You can read more about each of these commands in the main tools documentation (see README.md or https://nf-co.re/tools)
Tools helper code
- Fixed some bugs in the command line interface for
nf-core launchand improved formatting [#829] - New functionality for
nf-core downloadto make it compatible with DSL2 pipelines [#832]- Singularity images in module files are now discovered and fetched
- Direct downloads of Singularity images in python allowed (much faster than running
singularity pull) - Downloads now work with
$NXF_SINGULARITY_CACHEDIRso that pipelines sharing containers have efficient downloads
- Changed behaviour of
nf-core synccommand [#787]- Instead of opening or updating a PR from
TEMPLATEdirectly todev, a new branch is now created fromTEMPLATEand a PR opened from this todev. - This is to make it easier to fix merge conflicts without accidentally bringing the entire pipeline history back into the
TEMPLATEbranch (which makes subsequent sync merges much more difficult)
- Instead of opening or updating a PR from
Linting
- Major refactor and rewrite of pipieline linting code
- Much better code organisation and maintainability
- New automatically generated documentation using Sphinx
- Numerous new tests and functions, removal of some unnecessary tests
- Added lint check for merge markers [#321]
- Added new option
--fixto automatically correct some problems detected by linting - Added validation of default params to
nf-core schema lint[#823] - Added schema validation of GitHub action workflows to lint function [#795]
- Fixed bug in schema title and description validation
- Added second progress bar for conda dependencies lint check, as it can be slow [#299]
- Added new lint test to check files that should be unchanged from the pipeline.
- Added the possibility to ignore lint tests using a
nf-core-lint.ymlconfig file [#809]
1.12.1 - Silver Dolphin
Minor release, primarily to fix a problem introduced in v1.12 where not all instances of baseDir had been switched with projectDir. However, the elves have been busy and managed to squeeze in a few extra goodies along with the fix:
Template
8000- Finished switch from
$baseDirto$projectDiriniGenomes.confandmain.nf- Main fix is for
smail_fieldswhich was a bug introduced in the previous release. Sorry about that!
- Main fix is for
- Ported a number of small content tweaks from nf-core/eager to the template [#786]
- Better contributing documentation, more placeholders in documentation files, more relaxed markdownlint exceptions for certain HTML tags, more content for the PR and issue templates.
Tools helper code
- Pipeline schema: make parameters of type
rangetonumber. [#738] - Respect
$NXF_HOMEwhen looking for pipelines withnf-core list[#798] - Swapped PyInquirer with questionary for command line questions in
launch.py[#726]- This should fix conda installation issues that some people had been hitting
- The change also allows other improvements to the UI
- Fix linting crash when a file deleted but not yet staged in git [#796]
1.12 - Mercury Weasel
A bit of a fast cycle on this release, it is needed urgently because GitHub Actions CI tests will be failing on all pipelines due to the set-env and add-path commands being depreciated (see the GitHub blog post for details).
Tools helper code
- Updated
nf_coredocumentation generator for building https://nf-co.re/tools-docs/
Template
- Make CI comments work with PRs from forks [#765]
- Branch protection and linting results should now show on all PRs
- Updated GitHub issue templates, which had stopped working
- Refactored GitHub Actions so that the AWS full-scale tests are triggered after docker build is finished
- DockerHub push workflow split into two - one for dev, one for releases
- Updated actions to no longer use
set-envwhich is now depreciating [#739] - Added config import for
test_fullinnextflow.config - Switched depreciated
$baseDirto$projectDir - Updated minimum Nextflow version to
20.04.10 - Make Nextflow installation less verbose in GitHub Actions [#780]
Linting
- Updated code to display colours in GitHub Actions log output
- Allow tests to pass with
devversion of nf-core/tools (previous failure due to base image version) - Lint code no longer tries to post GitHub PR comments. This is now done in a GitHub Action only.
1.11 - Iron Tiger
Template
- Fix command error in
awstest.ymlGitHub Action workflow. - Allow manual triggering of AWS test GitHub Action workflows.
- Remove TODO item, which was proposing the usage of additional files beside
usage.mdandoutput.mdfor documentation. - Added a Podman profile, which enables Podman as container.
- Updated linting for GitHub actions AWS tests workflows.
Linting
- Made a base-level
Dockerfilea warning instead of failure - Added a lint failure if the old
bin/markdown_to_html.rscript is found - Update
richpackage dependency and use new markup escaping to change[[!]]back to[!]again
Other
- Pipeline sync - fetch full repo when checking out before sync
- Sync - Add GitHub actions manual trigger option
1.10.2 - Copper Camel [brought back from the dead]
Second patch release to address some small errors discovered in the pipeline template.
Apologies for the inconvenience.
- Fix syntax error in
/push_dockerhub.ymlGitHub Action workflow - Change
params.readPaths->params.input_pathsintest_full.config - Check results when posting the lint results as a GitHub comment
- This feature is unfortunately not possible when making PRs from forks outside of the nf-core organisation for now.
- More major refactoring of the automated pipeline sync
- New GitHub Actions matrix parallelisation of sync jobs across pipelines [#673]
- Removed the
--allbehaviour fromnf-core syncas we no longer need it - Sync now uses a new list of pipelines on the website which does not include archived pipelines [#712]
- When making a PR it checks if a PR already exists - if so it updates it [#710]
- More tests and code refactoring for more stable code. Hopefully fixes 404 error [#711]
1.10.1 - Copper Camel [patch]
Patch release to fix the automatic template synchronisation, which failed in the v1.10 release.
- Improved logging:
nf-core --log-file log.txtnow saves a verbose log to disk. - nf-core/tools GitHub Actions pipeline sync now uploads verbose log as an artifact.
- Sync - fixed several minor bugs, made logging less verbose.
- Python Rich library updated to
>=4.2.1 - Hopefully fix git config for pipeline sync so that commit comes from @nf-core-bot
- Fix sync auto-PR text indentation so that it doesn't all show as code
- Added explicit flag
--show-passedfornf-core lintinstead of taking logging verbosity