Error in checkForAndCreateDir

Hi,

I'm running DROP v1.4.0 on a set of 10 samples, with only the AberrantSplicing and AberrantExpression modules. The AberrantExpression module completes successfully, but I get the following error with the AberrantSplicing module:

[Thu Jun 27 09:57:03 2024]
rule AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R:
input: Output/processed_data/aberrant_splicing/datasets/savedObjects/raw-local-fraser/jaccard.h5, Scripts/AberrantSplicing/pipeline/Counting/03_filter_expression_FraseR.R
output: Output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/fds-object.RDS, Output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/filter_FRASER2.done
log: /mnt/disks/cromwell_root/.drop/tmp/AS/fraser/03_filter.Rds
jobid: 38
reason: Missing output files: Output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/filter_FRASER2.done; Input files updated by another job: Output/processed_data/aberrant_splicing/datasets/savedObjects/raw-local-fraser/jaccard.h5
wildcards: dataset=fraser
threads: 3
resources: tmpdir=/mnt/disks/cromwell_root/tmp.8ef94c54

Load packages
Loading assay: rawCountsJ
Loading assay: psi5
Loading assay: psi3
Loading assay: rawOtherCounts_psi5
Loading assay: rawOtherCounts_psi3
Loading assay: rawCountsJnonsplit
Loading assay: jaccard
Loading assay: rawOtherCounts_jaccard
Loading assay: delta_jaccard
Loading assay: delta_psi5
Loading assay: delta_psi3
Loading assay: rawCountsSS
Loading assay: theta
Loading assay: rawOtherCounts_theta
Loading assay: delta_theta
symLink fraser dir
Thu Jun 27 09:57:24 2024: Filtering out introns with low read support ...
Thu Jun 27 09:57:49 2024: Filtering out non-variable introns ...
Thu Jun 27 09:57:53 2024: Filtering done!
Error in checkForAndCreateDir(fds, outDir) :
Can not create workding directory: Output/processed_data/aberrant_splicing/datasets//savedObjects/raw-fraser
Calls: saveFraserDataSet -> checkForAndCreateDir
In addition: Warning message:
In dir.create(dir, recursive = TRUE) :
'Output/processed_data/aberrant_splicing/datasets//savedObjects/raw-fraser' already exists
Execution halted
[Thu Jun 27 09:57:54 2024]
Error in rule AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R:
jobid: 38
input: Output/processed_data/aberrant_splicing/datasets/savedObjects/raw-local-fraser/jaccard.h5, Scripts/AberrantSplicing/pipeline/Counting/03_filter_expression_FraseR.R
output: Output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/fds-object.RDS, Output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/filter_FRASER2.done
log: /mnt/disks/cromwell_root/.drop/tmp/AS/fraser/03_filter.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /mnt/disks/cromwell_root/tmp.8ef94c54/tmprhayda0h, line 183:
Command 'set -euo pipefail; Rscript --vanilla /mnt/disks/cromwell_root/.snakemake/scripts/tmpcb4w4ev7.03_filter_expression_FraseR.R' returned non-zero exit status 1.
File "/mnt/disks/cromwell_root/tmp.8ef94c54/tmprhayda0h", line 183, in __rule_AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R
File "/root/miniconda3/envs/DROP_1.4.0/lib/python3.12/concurrent/futures/thread.py", line 58, in run
Select jobs to execute...

This is what I have in my config file:

projectTitle: Detection of RNA Outlier Pipeline - VAL1-10, AS, AE
htmlOutputPath: Output/html
indexWithFolderName: true
root: Output
sampleAnnotation: sample_annotation_VAL1-10.tsv
geneAnnotation:
v44: gencode.v44.annotation.gtf
genomeAssembly: hg38
genome:
gatk: GRCh38_full_analysis_set_plus_decoy_hla.fa
hpoFile: null
random_seed: false
exportCounts:
geneAnnotations:

v44

aberrantExpression:
run: true
groups:

outrider
fpkmCutoff: 1
implementation: autoencoder
padjCutoff: 0.05
zScoreCutoff: 0
genesToTest: null
maxTestedDimensionProportion: 3
yieldSize: 2000000

aberrantSplicing:
run: true
groups:

fraser
recount: true
longRead: false
keepNonStandardChrs: false
filter: true
minExpressionInOneSample: 20
quantileMinExpression: 10
minDeltaPsi: 0.05
implementation: PCA
padjCutoff: 0.1
maxTestedDimensionProportion: 6
genesToTest: null
FRASER_version: FRASER2
deltaPsiCutoff : 0.1
quantileForFiltering: 0.75

And this is the snakemake command: snakemake --cores 3 -k

Please let me know if there's something that needs to be updated in the config file. Thanks!

Hi, hard to say, the error returned non-zero exit status 1 usually appears when there's not enough memory allocated. Can you please check that? Also, you may want to change recount: false, otherwise your samples get recounted every time you execute the aberrantSplicing rule. Finally, add the --rerun-triggers mtime parameter every time you run the aberrantSplicing module

Hi, thanks for the prompt response. I believe I am providing sufficient memory for the run - I'm running DROP on a machine with 256GB memory and 30cores. I re-ran the same set of 10 samples with recount: false and --re-run-triggers mtime, and still get the same error.
I manually ran the 03_filter_expression_FraseR.R script and ran into the same error. This specific line throws the error:
devNull <- saveFraserDataSet(fds,dir = workingDir)

Hi, can you check what the output of dir.exists(workingDir) is for your case? It seems there is an issue in recognizing that the directory already exists for some reason.
Also, what version of FRASER are you using?

Hi,
This is the FRASER version: FRASER_1.99.4
and the directory does exist:
dir.exists("Output/processed_data/aberrant_splicing/datasets/") [1] TRUE

Also, I should add that when I run the same module on a single sample with external counts from 10 other samples, this AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R script runs without error.

I believe I am running into the same error (DROP version 1.3.3). I have managed to run through the splicing module previously when using external counts. In this run I only have a set of ~100 non-external samples. The AE module went through without external counts.

Looking in htop while running, it looks like there is plenty of RAM available so I don't think this is a memory issue.

A bit up in the R script I see that it is symlinking if no external counts are present. Can it be related to this? Seems saveFraserDataSet in the FRASER package somehow collides with this.

# Add external data if provided by dataset
if(length(exCountIDs) > 0){
  ...
} else {
    message("symLink fraser dir")
    file.symlink(paste0(workingDir, "savedObjects/","raw-local-", dataset),
                 paste0(workingDir, "savedObjects/","raw-", dataset))
    
    fds@colData$isExternal <- as.factor(FALSE)
    workingDir(fds) <- workingDir
    name(fds) <- paste0("raw-", dataset)
}

The singularity container I am using to run DROP uses FRASER_1.99.3.

Here is my stack trace:

Fri Aug 16 12:18:54 2024: Filtering out introns with low read support ...
Fri Aug 16 12:20:38 2024: Filtering out non-variable introns ...
Fri Aug 16 12:21:10 2024: Filtering done!
Error in checkForAndCreateDir(fds, outDir) :
  Can not create workding directory: output/processed_data/aberrant_splicing/datasets//savedObjects/raw-fraser
Calls: saveFraserDataSet -> checkForAndCreateDir
In addition: Warning message:
In dir.create(dir, recursive = TRUE) :
  'output/processed_data/aberrant_splicing/datasets//savedObjects/raw-fraser' already exists
Execution halted
[Fri Aug 16 12:21:11 2024]
Error in rule AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R:
    jobid: 3
    input: output/processed_data/aberrant_splicing/datasets/savedObjects/raw-local-fraser/jaccard.h5, Scripts/AberrantSplicing/pipeline/Counting/03_filter_expression_FraseR.R
    output: output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/fds-object.RDS, output/processed_data/aberrant_splicing/datasets/savedObjects/fraser/filter_FRASER2.done
    log: <work>/.drop/tmp/AS/fraser/03_filter.Rds (check log file(s) for error details)

RuleException:
CalledProcessError in file /tmp/tmpuq_thm5b, line 89:
Command 'set -euo pipefail;  Rscript --vanilla <work>/.snakemake/scripts/tmporwzmnov.03_filter_expression_FraseR.R' returned non-zero exit status 1.
  File "/tmp/tmpuq_thm5b", line 89, in __rule_AberrantSplicing_pipeline_Counting_03_filter_expression_FraseR_R
  File "/opt/conda/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-08-16T121828.769496.snakemake.log

Any thoughts on this one? This is the one step blocking us at the moment from evaluating the DROP output in our data.

@Jakob37

Hi @Jakob37,

I cannot reproduce this issue and haven't seen it before. My guess is that if you remove the directory (output/processed_data/aberrant_splicing/datasets/savedObjects/raw-fraser) and re-run the pipeline, it will run successfully.

@Jakob37

Hi @Jakob37,

I cannot reproduce this issue and haven't seen it before. My guess is that if you remove the directory (output/processed_data/aberrant_splicing/datasets/savedObjects/raw-fraser) and re-run the pipeline, it will run 8000 successfully.

Hello @AtaJadidAhari and thank you for taking the time to look into this.

I digged into this a bit more. I could indeed reproduce the error, and also tried removing the soft-link as you mentioned, which did not solve it.

The issue seems to be that the soft link to the raw-local-fraser folder is soft-linked using the wrong working directory. This yields an invalid link as such:

<workdir>/output/processed_data/aberrant_splicing/datasets/savedObjects $ ls -l
lrwxrwxrwx 1 jakob cmd-bnf 78 Aug 22 20:31 raw-fraser -> output/processed_data/aberrant_splicing/datasets/savedObjects/raw-local-fraser
drwxr-xr-x 3 jakob cmd-bnf 19 Aug 22 20:31 raw-local-fraser

This causes the crash as such:

saveFraserDataset in saveHDF5Objects.R is called, which in turn calls checkForAndCreateDir in helper-functions.R
Here, the dir is now a symlink pointing to an empty location.
dir.exists(dir) will yield FALSE on the broken symlink, but will fail to create the dir with a warning.
Finally the "Can not create workding directory" message is triggered from the code below.

checkForAndCreateDir <- function(object, dir){
    verbose <- 0
    if(is(object, "FraserDataSet")){
        verbose <- verbose(object)
        if(missing(dir)){
            dir <- workingDir(object)
        }
    }
    if(!dir.exists(dir)){
        if(verbose > 1){
            message(date(), ": The given working directory '", 
                    dir, "' does not exists. We will create it.")
        }
        dir.create(dir, recursive=TRUE)
    }
    if(!dir.exists(dir)){
        stop("Can not create workding directory: ", dir)
    }
    return(TRUE)
}

I tried hard-coding in the absolute path for the work dir into the DROP-script, and then the step successfully went through.

Unsure why it sometimes works then, as it works for you. Might be a version issue? (Edit: I see now that the author of the issue has the same issue with the latest version). Or something with running it through SLURM. I am using a container running DROP 1.3.3 (running this through the nextflow pipeline Tomte).

OK, after some more digging I think I have figured out the issue.

It seems the current code will only work if provided an absolute path. If given a relative path, it will crash here.

The working dir is calculated as such:

#'   - workingDir: '`sm cfg.getProcessedDataDir() + "/aberrant_splicing/datasets/"`'

The cfg.getProcessedDataDir() value is assign as such:

self.processedDataDir = self.root / "processed_data"

I suspect self.root comes from the config.yaml file.

At the moment (as generated by the Nextflow pipeline I am running), it is provided as such, i.e. with a relative path:

root: output

I think we could fix this in the Tomte pipeline, such that this isn't an issue.

It would be helpful though with an early error for relative paths if not allowed.

@Jakob37

Thanks @Jakob37 for digging out! We'll now see on our side how to prevent this.

@Jakob37

Thanks @Jakob37 for looking deeper into this issue!
@AtaJadidAhari do you have any updates on when this will be fixed? Thanks!

@Jakob37

Hi @Jakob37, thanks we know how to fix the issue with the relative path in the root. We will evaluate if it's an absolute or relative path and throw an error if it's relative right at the start.
@pkrithivasan, sorry, I'm not sure what is the issue in your case :) do you have an absolute or relative path in the root parameter in the config file?

Navigation Menu

Repository navigation

Error in checkForAndCreateDir #558

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Error in checkForAndCreateDir #558

Description

Activity

vyepez88 commented on Jul 1, 2024

pkrithivasan commented on Jul 3, 2024

ischeller commented on Jul 4, 2024

pkrithivasan commented 8000 on Jul 16, 2024

pkrithivasan commented on Jul 16, 2024

Jakob37 commented on Aug 16, 2024

Jakob37 commented on Aug 23, 2024

AtaJadidAhari commented on Aug 23, 2024

Jakob37 commented on Aug 28, 2024

Jakob37 commented on Aug 28, 2024

vyepez88 commented on Aug 28, 2024

pkrithivasan commented on Nov 18, 2024

vyepez88 commented on Feb 18, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions