Ttyjfyk

This is an extension of the question I asked .
Passing directory from command line to shell script

I have a script which writes another script using a heredoc. I need to be able to write unexpanded variables in the heredoc, so I use single quotes ('EOF'). However, I need one variable to be expanded. Given the script below, how can I write the value of $sourcedir inside the heredoc?

#!/bin/bash 

sourcedir="$1"
cd $sourcedir

find "$PWD" -maxdepth 2 -name *_R1*.fastq.gz > list1

fastq_list=$sourcedir/list1 echo `cat $fastq_list` num_files=$(wc -l < 
 $sourcedir/list1) echo $num_files



cat > run_array_job.sh<<'EOF'

#!/bin/bash -l
#$ -j y
#$ -cwd -S /bin/sh
#$ -l h_vmem=10G
#$ -pe smp 12

if [ -z "$SGE_TASK_ID" ]; then echo "Need to set SGE_TASK_ID" exit 1 fi


 BASEDIR=$sourcedir

 echo "BASEDIR" echo $BASEDIR

 BASEFILES=$( ls *_R1.fastq.gz)
 BASEFILES_ARRAY=($BASEFILES)
 BASEFILE=$BASEFILES_ARRAY[($SGE_TASK_ID - 1)]
 echo $BASEFILE


 ...................
 ...................

 EOF

 qsub -t 1-$num_files run_array_job.sh

I am running this script using

bash script.sh /home/dir/data

I am able to pass /home/dir/data as $1 to sourcedir but it also needs to be passed to BASEDIR , in array script which is submitted to cluster using qsub.

You can most straightforwardly do this for your usage just by breaking your heredoc in two parts:

cat > run_array_job.sh<<'EOF'

#!/bin/bash -l
...
EOF

printf 'BASEDIR="%s"n" "$sourcedir" >> run_array_job.sh

cat >> run_array_job.sh<<'EOF'
echo "BASEDIR" echo $BASEDIR
...
EOF

qsub -t 1-$num_files run_array_job.sh

This just builds the first part of the file, appends the single line that you wanted the variable available for at the end of that first part using >>, and then joins the rest of the document on the end in the same way.

You end up with the same coherent file at the end, and only write a couple of lines extra. If you have multiple variables to pass through, you can put them all in at once as well.

That's somewhat hard to do, since unlike with quotes, you can't just stop and restart expansion in an here-doc. But we could post-process the here-doc with sed:

#!/bin/bash
sourcedir=/some/path
sed -e "s,%%sourcedir%%,$sourcedir,g" << 'EOF'
some commands here with $variables not expanded 
except for the special %%sourcedir%%, which is
EOF

Running that produces the output:

some commands here with $variables not expanded 
except for the special /some/path, which is

The sed command simply changes all instances of %%sourcedir%% with whatever the value of $sourcedir (as long as it doesn't contain commas; you'd need to change the separator for the s command to something else then.)

I changed the placeholder to another format for clarity's sake, but you could also leave it as $sourcedir and then use sed -e "s,$sourcedir,$sourcedir,". (Though note that it would also match $sourcedirectory and other similar variables, but not $sourcedir even though that would be equivalent in the shell.)

Alternatively, use GNU envsubst on the document, if you have it (it's part of gettext):

#!/bin/bash
export sourcedir=/some/path
envsubst '$sourcedir' << 'EOF'
some commands here with $variables not expanded 
except for the special $sourcedir, which is
EOF