Sinto filterbarcodes completes without error but sometimes temporary files remain and a few reads are lost

[jenellewallace]

Hello,
Thanks for your helpful software! I have an issue with running filterbarcodes on an SGE cluster where sometimes (maybe 10% of runs) temporary files are left in my output folder and there is a small difference in the number of reads included in the final files for successful runs vs the ones where temporary files remain. I first noticed this in cases with multiple cores but have since reproduced this issue with a single core. The temporary files all have 0 size and there is one for each group per core.

Here is some example output:

Function run_filterbarcodes called with the following arguments:

bam     merged_atac.bam
cells   metadata.tsv
trim_suffix     False
nproc   1
barcode_regex   None
barcodetag	CB
outdir  split_check/atac
sam     False
func    <function run_filterbarcodes at 0x1552933f5ea0>

Function completed in  685.0 m 10.89 s

Here is what the output folder looks like after the run:

total 26378415
-rw-r--r--. 1 jwallace root       7084 Jun 27 06:18 Astro_H103.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 Astro_H103_FRIFFN
-rw-r--r--. 1 jwallace root       7084 Jun 27 05:23 Astro_H106.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 Astro_H106_FRIFFN
-rw-r--r--. 1 jwallace root  553288605 Jun 27 06:18 Astro_H11.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 Astro_H11_FRIFFN
-rw-r--r--. 1 jwallace root  492693629 Jun 27 06:21 Astro_H15.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 Astro_H15_FRIFFN
-rw-r--r--. 1 jwallace root  659880539 Jun 27 05:32 Astro_H24_prelim.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 Astro_H24_prelim_FRIFFN
-rw-r--r--. 1 jwallace root 1617216265 Jun 27 05:30 Astro_H25_prelim.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 Astro_H25_prelim_FRIFFN
-rw-r--r--. 1 jwallace root       7084 Jun 27 05:23 Astro_H99.bam
-rw-r--r--. 1 jwallace root       6632 Jun 25 08:40 Astro_H99_FRIFFN
-rw-r--r--. 1 jwallace root       7084 Jun 27 05:23 EN_H103.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 EN_H103_FRIFFN
-rw-r--r--. 1 jwallace root       7084 Jun 27 06:54 EN_H106.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 EN_H106_FRIFFN
-rw-r--r--. 1 jwallace root  565896150 Jun 27 05:34 EN_H11.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 EN_H11_FRIFFN
-rw-r--r--. 1 jwallace root  693521254 Jun 27 06:20 EN_H15.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 EN_H15_FRIFFN
-rw-r--r--. 1 jwallace root  976355302 Jun 27 05:23 EN_H24_prelim.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 EN_H24_prelim_FRIFFN
-rw-r--r--. 1 jwallace root  387659539 Jun 27 05:05 EN_H25_prelim.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 EN_H25_prelim_FRIFFN
-rw-r--r--. 1 jwallace root       7084 Jun 27 05:34 EN_H99.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 EN_H99_FRIFFN
-rw-r--r--. 1 jwallace root       7084 Jun 27 05:30 IN_dLGE_CGE_H103.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 IN_dLGE_CGE_H103_FRIFFN
-rw-r--r--. 1 jwallace root       7084 Jun 27 06:18 IN_dLGE_CGE_H106.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 IN_dLGE_CGE_H106_FRIFFN
-rw-r--r--. 1 jwallace root 6083539873 Jun 27 06:54 IN_dLGE_CGE_H11.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 IN_dLGE_CGE_H11_FRIFFN
-rw-r--r--. 1 jwallace root 7170691475 Jun 27 06:03 IN_dLGE_CGE_H15.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 IN_dLGE_CGE_H15_FRIFFN
-rw-r--r--. 1 jwallace root 2544213263 Jun 27 06:16 IN_dLGE_CGE_H24_prelim.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 IN_dLGE_CGE_H24_prelim_FRIFFN
-rw-r--r--. 1 jwallace root 6709875593 Jun 27 05:21 IN_dLGE_CGE_H25_prelim.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 IN_dLGE_CGE_H25_prelim_FRIFFN
-rw-r--r--. 1 jwallace root       7084 Jun 27 05:23 IN_dLGE_CGE_H99.bam
-rw-r--r--. 1 jwallace root          0 Jun 25 08:40 IN_dLGE_CGE_H99_FRIFFN

Do you have any suggestions to prevent this issue? Thanks!

[timoast]

I'll try to look into this but in the meantime you could try using the subset-bam function from 10x instead: https://github.com/10XGenomics/subset-bam

[guptas2005]

@jenellewallace This is unrelated, but how did you get Sinto to output multiple BAM files on the same run? Does your metafile contain just one large list of barcodes along with cell types? Does Sinto automatically know how to create different BAM files for different cell types?