• use a date-based directory structure for the experiments you perform
• keep a lab-notebook containing documentation and code for each experiment (How about Org-mode?)
• create generate scripts that work on original data and a general runner script for the complete experiment

His work flow is applies mostly to *nix enviroments. One thing I’d like to add: Use symbolic links! Due to huge amounts of data and backup strategies, it might be impractical to have all the project’s data in a single directory. Create subdirectories of your data and result directories according to the naming convention and link them to other network drives using the ‘ln -s’ command.

#!/bin/bash
paramset="foo.txt 1 --with-graphics
bar.txt 8 --no-graphics
flock.txt 4 --with-graphics"

echo "$du" | while read file p1 p2 ; do
./myProgram $file -t $p1 --verbose $p2
done

We here run the program myProgram three times (for each line in the multi-line string). Every line contains three white-space separated values (words), to which we assign the variable names file, p1, p2 in the loop header. Note that the last variable (in this case p2) always contains all remaining words of a given line if there are more words then variables.

The set of parameters can also be stored in a file. In that case, replace the loop header with cat params.txt | while read file p1 p2 ; do. If the script is not working properly, examine the Input-Field-Separator (IFS) variable, which should be set to IFS=" ".