Repository navigation
countReads.R error #676
Description
I've been trying to run the aberrant expression and aberrant splicing modules on STAR mapped data, and keep running into this error:
Error in rule AberrantExpression_pipeline_Counting_countReads_R:
jobid: 18
input: /well/bsg/projects/BIOINF1_Taylor/pipeline/working/STAR_sorted/D2wt4.Sorted.out.bam, /gpfs3/well/bsg/projects/BIOINF1_Taylor/drop/Output/processed_data/aberrant_expression/GRCh38.113/count_ranges.Rds, /gpfs3/well/bsg/projects/BIOINF1_Taylor/drop/Output/processed_data/aberrant_expression/GRCh38.113/params/counts/D2wt4_countParams.csv, Scripts/AberrantExpression/pipeline/Counting/countReads.R
output: /gpfs3/well/bsg/projects/BIOINF1_Taylor/drop/Output/processed_data/aberrant_expression/GRCh38.113/counts/D2wt4.Rds
log: /gpfs3/well/bsg/projects/BIOINF1_Taylor/drop/.drop/tmp/AE/GRCh38.113/counts/D2wt4.Rds (check log file(s) for error details)
RuleException:
CalledProcessError in file /tmp/tmp_vut99tg, line 35:
Command 'set -euo pipefail; Rscript --vanilla /gpfs3/well/bsg/projects/BIOINF1_Taylor/drop/.snakemake/scripts/tmpt5lqlfo2.countReads.R' returned non-zero exit status 1.
File "/tmp/tmp_vut99tg", line 35, in __rule_AberrantExpression_pipeline_Counting_countReads_R
File "/well/bsg/users/bjw/conda-envs/drop/lib/python3.11/concurrent/futures/thread.py", line 58, in run
I'd seen in another issue that this can be related to memory; but here I'm running with --cores 16 in a SLURM interactive session with 60GB of RAM available. From the logs, it looks like the biggest jobs are for 6 cores and 22GB of RAM, so I would have thought this would fit comfortably. I've not been able to find any information as to exactly what the error was; a missing R script file, an error in the R script, a failure to run the script, or something else.
I've made sure that DROP is fully updated, as per bioconda.
Full log (I ran with -k to get what I could):
Activity
AtaJadidAhari commentedon Jul 31, 2026
hi @Queex ,
My first guess is that it is still a memory issue. Although you have 60GB available in total, your jobs are only claiming ~22GB, which is not enough for some samples. You should increase the memory for the jobs, not just your slurm session. You can use
--default-resources mem_mb=40000flag in your snakemake command to increase the memory of each job.Let us know if it solves the issue or not.
Queex commentedon Jul 31, 2026
@AtaJadidAhari Thanks for the help, although it doesn't seem to have fixed it yet.
I'm getting the same error. The resources listed for the job now say
mem_mb=22000, mem_mib=20981, disk_mb=22647, disk_mib=21598, so it looks like it's having to use swap to meet the increased memory allocation. This might be because I'm specifying total RAM to slurm and not setting RAM per CPU, but I would have thought it would still run, just more slowly.I was feeling suspicious that something might have been going awry earlier, so I just tried running with
-Fto make sure the log included everything from scratch. There seem to be some cluster issues; as this is the fourth time an interactive session has hung and forced me to cancel the job and try again. I've attached the log for that run while wait for the next attempt to queue up again.2026-07-31T133705.415514.snakemake.log
For some reason, that run also created two additional, empty log files. The job was split over three physical machines, so that might be why.
Queex commentedon Jul 31, 2026
Still no joy. Same error as before, although this time I noticed that the output to the screen contains lines that do not appear in the log, attached separately. Not sure why that would be causing problems.
2026-07-31T142128.414255.snakemake.log
screen_output.txt
One thing that occurred to me - I want results for the entire genome, rather than an a priori list of genes; to that end I commented out the genesToTest setting in
config.yaml. I was expecting an immediate error if the list is a strict requirement, when there wasn't one I assumed that there would be output without the FDR list. If that's a strict requirement, presumably I should process the gtf file to dump a list of all gene names.ETA: I see from running
drop initin a new test folder that genesToTest can be set tonull; in this new test folder I'll try that.Queex commentedon Aug 3, 2026
Further peculiarities:
--cores 3. This seems to imply that that trying to assign more memory and slurm using disk instead still causes the problem, but of course that it very slow.It might make more sense to run with
--slurm, but I feel like I'm committed now to let this finish as-is so I can triage any data/config issues.Queex commentedon Aug 7, 2026
Okay, this issue recurs even when using
--slurm, and it looks as though snakemake is ignoring the--default-resources mem_mbargument. Despite invoking with--slurm --jobs 2 --default-resources mem_mb=256000 tasks=6, the log still says:for each job.