NAME BioX::Workflow::Plugin::Drake - A very opinionated template based bioinformatics workflow writer for Drake. SYNOPSIS The main documentation for this module is at BioX::Workflow. This module extends Workflow in order to add functionality for outputing workflows in drake format. biox-workflow-drake.pl --workflow workflow.yml > workflow.drake drake --workflow workflow.drake #with other functionality such as --jobs for asynchronous output, etc. More information about Drake can be found here https://github.com/Factual/drake. Default Variables BioX::Workflow::Plugin::Drake assumes your INPUT/OUTPUT and indir/outdirs are linked. This means the output from step1 is the input for step2. You can override this behavior by either declaring any of these values, or in the global variables set auto_input: 0, disable automatic indir/outdir naming with auto_name: 0, and disable automatically naming outdirectories by rule names with enforce_struct: 0. Example workflow.yml --- global: - indir: /home/user/workflow - outdir: /home/user/workflow/output - file_rule: (.csv)$ rules: - backup: local: - INPUT: "{$self->indir}/{$sample}.csv" - OUTPUT: "{$self->outdir}/{$sample}.csv" - thing: "other thing" process: | cp $INPUT $OUTPUT - grep_VARA: local: - OUTPUT: "{$self->outdir}/{$sample}.grep_VARA.csv" process: | echo "Working on {$self->{indir}}/{$sample.csv}" grep -i "VARA" {$self->indir}/{$sample}.csv >> {$self->outdir}/{$sample}.grep_VARA.csv \ || touch {$self->OUTPUT} - grep_VARB: local: - OUTPUT: "{$self->outdir}/{$sample}.grep_VARA.grep_VARB.csv" process: | grep -i "VARB" {$self->indir}/{$sample}.grep_VARA.csv >> {$self->outdir}/{$sample}.grep_VARA.grep_VARB.csv || touch {$self->OUTPUT} Notes on the drake.yml Drake will stop everything if you're job returns with an exit code of anything besides 0. For this reason we have the last command have a command1 || command2 syntax, so that even if we don't grep any "VARB" from the file the workflow could continue. Run it with default setup biox-workflow-drake.pl --workflow workflow.yml > workflow.full.drake Output with default setup I don't want to inlcude the whole file, but you get the idea ; ; Generated at: 2015-06-21T11:01:24 ; This file was generated with the following options ; --workflow drake.yml ; --min 1 ; ; ; Samples: test1, test2 ; ; ; Starting Workflow ; ; ; Starting backup ; ; ; Variables ; Indir: /home/guests/jir2004/workflow ; Outdir: /home/guests/jir2004/workflow/output/backup ; Local Variables: ; INPUT: {$self->indir}/{$sample}.csv ; OUTPUT: {$self->outdir}/{$sample}.csv ; thing: other thing ; /home/guests/jir2004/workflow/output/backup/$[SAMPLE].csv <- /home/guests/jir2004/workflow/$[SAMPLE].csv cp $INPUT $OUTPUT ; ; Ending backup ; ; ; Starting grep_VARA ; Run drake drake --workflow workflow.full.drake The following steps will be run, in order: 1: /home/user/workflow/output/backup/test1.csv <- /home/user/workflow/test1.csv [timestamped] 2: /home/user/workflow/output/backup/test2.csv <- /home/user/workflow/test2.csv [timestamped] 3: /home/user/workflow/output/grep_vara/test1.grep_VARA.csv <- /home/user/workflow/output/backup/test1.csv [projected timestamped] 4: /home/user/workflow/output/grep_vara/test2.grep_VARA.csv <- /home/user/workflow/output/backup/test2.csv [projected timestamped] 5: /home/user/workflow/output/grep_varb/test1.grep_VARA.grep_VARB.csv <- /home/user/workflow/output/grep_vara/test1.grep_VARA.csv [projected timestamped] 6: /home/user/workflow/output/grep_varb/test2.grep_VARA.grep_VARB.csv <- /home/user/workflow/output/grep_vara/test2.grep_VARA.csv [projected timestamped] Confirm? [y/n] y Running 6 steps with concurrence of 1... --- 0. Running (timestamped): /home/user/workflow/output/backup/test1.csv <- /home/user/workflow/test1.csv --- 0: /home/user/workflow/output/backup/test1.csv <- /home/user/workflow/test1.csv -> done in 0.02s --- 1. Running (timestamped): /home/user/workflow/output/backup/test2.csv <- /home/user/workflow/test2.csv --- 1: /home/user/workflow/output/backup/test2.csv <- /home/user/workflow/test2.csv -> done in 0.01s --- 2. Running (timestamped): /home/user/workflow/output/grep_vara/test1.grep_VARA.csv <- /home/user/workflow/output/backup/test1.csv Working on /home/user/workflow/output/backup/test1csv --- 2: /home/user/workflow/output/grep_vara/test1.grep_VARA.csv <- /home/user/workflow/output/backup/test1.csv -> done in 0.01s --- 3. Running (timestamped): /home/user/workflow/output/grep_vara/test2.grep_VARA.csv <- /home/user/workflow/output/backup/test2.csv Working on /home/user/workflow/output/backup/test2csv --- 3: /home/user/workflow/output/grep_vara/test2.grep_VARA.csv <- /home/user/workflow/output/backup/test2.csv -> done in 0.01s --- 4. Running (timestamped): /home/user/workflow/output/grep_varb/test1.grep_VARA.grep_VARB.csv <- /home/user/workflow/output/grep_vara/test1.grep_VARA.csv --- 4: /home/user/workflow/output/grep_varb/test1.grep_VARA.grep_VARB.csv <- /home/user/workflow/output/grep_vara/test1.grep_VARA.csv -> done in 0.01s --- 5. Running (timestamped): /home/user/workflow/output/grep_varb/test2.grep_VARA.grep_VARB.csv <- /home/user/workflow/output/grep_vara/test2.grep_VARA.csv --- 5: /home/user/workflow/output/grep_varb/test2.grep_VARA.grep_VARB.csv <- /home/user/workflow/output/grep_vara/test2.grep_VARA.csv -> done in 0.08s Done (6 steps run). Run in minified mode As an alternative you can run this with the --min option, which instead of printing out each workflow prints out only one, and creates a run-workflow.sh which has all of your environmental variables. This option is preferable if running on an HPC cluster with many nodes. This WILL break with use of --resample, either local or global. You need to split up your workflows as opposed to using the --resample option. biox-workflow-drake.pl --workflow workflow.yml --min 1 > workflow.drake #This also creates the run-workflow.sh in the same directory ./run-workflow.sh cat drake.log #Here is the log for the first run 2015-06-21 14:02:47,543 INFO Running 3 steps with concurrence of 1... 2015-06-21 14:02:47,568 INFO 2015-06-21 14:02:47,570 INFO --- 0. Running (timestamped): /home/user/workflow/output/backup/test1.csv <- /home/user/workflow/test1.csv 2015-06-21 14:02:47,592 INFO --- 0: /home/user/workflow/output/backup/test1.csv <- /home/user/workflow/test1.csv -> done in 0.02s #So on and so forth If you look in the example directory you will see a few png files, these are outputs of the drake workflow. =cut Acknowledgements Before version 0.03 This module was originally developed at and for Weill Cornell Medical College in Qatar within ITS Advanced Computing Team. With approval from WCMC-Q, this information was generalized and put on github, for which the authors would like to express their gratitude. As of version 0.03: This modules continuing development is supported by NYU Abu Dhabi in the Center for Genomics and Systems Biology. With approval from NYUAD, this information was generalized and put on bitbucket, for which the authors would like to express their gratitude. Inline Code Documentation You shouldn't need these, but if you do here they are. Attributes full Print the whole workflow hardcoded. This is the default min Print the workflow as 2 files. Run the drake things drake --vars "SAMPLE=$sample" --workflow/workflow.drake workflow.drake Our regular file Subroutines Subroutines before run Must initialize some variables after get_samples Things to do if we decide to do a min version write_process Fill in the template with the process Ensure INPUT/OUTPUT exist Prettyify the output a bit AUTHOR Jillian Rowe COPYRIGHT Copyright 2015- Jillian Rowe LICENSE This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. SEE ALSO