here-document basename conflicts?Calling multiple bash scripts and running them in parallel, not in sequenceHow to run x instances of a script parallel?Expanding only certain variables inside a heredocBash shell scripting basic question regarding the syntax and basenameBasename gives extra operand erroruse basename in find -exec?basename doesn't work as expectedHere Document on Platform LSFdirname and basename vs parameter expansionEscape a variable for use as content of another scriptHow to escape $ in here-documentConcatenate pwd and basenamebasename extra operand error
Are tax years 2016 & 2017 back taxes deductible for tax year 2018?
How to make payment on the internet without leaving a money trail?
Simulate Bitwise Cyclic Tag
Concept of linear mappings are confusing me
What is the offset in a seaplane's hull?
Banach space and Hilbert space topology
How can bays and straits be determined in a procedurally generated map?
What defenses are there against being summoned by the Gate spell?
How is the claim "I am in New York only if I am in America" the same as "If I am in New York, then I am in America?
Why doesn't Newton's third law mean a person bounces back to where they started when they hit the ground?
Is it possible to make sharp wind that can cut stuff from afar?
Why is "Reports" in sentence down without "The"
Can Medicine checks be used, with decent rolls, to completely mitigate the risk of death from ongoing damage?
A newer friend of my brother's gave him a load of baseball cards that are supposedly extremely valuable. Is this a scam?
whey we use polarized capacitor?
A Journey Through Space and Time
How does one intimidate enemies without having the capacity for violence?
I’m planning on buying a laser printer but concerned about the life cycle of toner in the machine
Can I interfere when another PC is about to be attacked?
Modification to Chariots for Heavy Cavalry Analogue for 4-armed race
If Manufacturer spice model and Datasheet give different values which should I use?
Why are 150k or 200k jobs considered good when there are 300k+ births a month?
Email Account under attack (really) - anything I can do?
N.B. ligature in Latex
here-document basename conflicts?
Calling multiple bash scripts and running them in parallel, not in sequenceHow to run x instances of a script parallel?Expanding only certain variables inside a heredocBash shell scripting basic question regarding the syntax and basenameBasename gives extra operand erroruse basename in find -exec?basename doesn't work as expectedHere Document on Platform LSFdirname and basename vs parameter expansionEscape a variable for use as content of another scriptHow to escape $ in here-documentConcatenate pwd and basenamebasename extra operand error
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I am using EOF to generate bash scripts that run the Rscripts. In the Rscript I used basename
to specify the output file name.
When I use EOF
to generate a list of bash scripts, I could not get basename
to work. The error message is shown below. I was still able to get the bash scripts generated but the $AF
turned into a blank in both places where it presented. Very strange!
I had the bash script tested and it is working so I know the problem is somewhere between EOF
and basename
.
How can I use basename
with EOF
? Or is there any alternative methods? Thank you.
for letter in A..Z
do cat <<- EOF > batch_$letter.sh
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
for AF in $BASE/$letter*.txt_step3; do
Rscript $R_funcP_tools.R
--ptool $R_func/P_tools_linux
--group $AF
--treat $TREAT/pop_exclude24dup
--out $OUTPUT/OUT_$(basename $AF%%_txt_step3)_noregress
--binary-target F; done
EOF
done
This is the error message
basename: missing operand Try 'basename --help' for more information.
bash scripting variable basename
add a comment |
I am using EOF to generate bash scripts that run the Rscripts. In the Rscript I used basename
to specify the output file name.
When I use EOF
to generate a list of bash scripts, I could not get basename
to work. The error message is shown below. I was still able to get the bash scripts generated but the $AF
turned into a blank in both places where it presented. Very strange!
I had the bash script tested and it is working so I know the problem is somewhere between EOF
and basename
.
How can I use basename
with EOF
? Or is there any alternative methods? Thank you.
for letter in A..Z
do cat <<- EOF > batch_$letter.sh
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
for AF in $BASE/$letter*.txt_step3; do
Rscript $R_funcP_tools.R
--ptool $R_func/P_tools_linux
--group $AF
--treat $TREAT/pop_exclude24dup
--out $OUTPUT/OUT_$(basename $AF%%_txt_step3)_noregress
--binary-target F; done
EOF
done
This is the error message
basename: missing operand Try 'basename --help' for more information.
bash scripting variable basename
add a comment |
I am using EOF to generate bash scripts that run the Rscripts. In the Rscript I used basename
to specify the output file name.
When I use EOF
to generate a list of bash scripts, I could not get basename
to work. The error message is shown below. I was still able to get the bash scripts generated but the $AF
turned into a blank in both places where it presented. Very strange!
I had the bash script tested and it is working so I know the problem is somewhere between EOF
and basename
.
How can I use basename
with EOF
? Or is there any alternative methods? Thank you.
for letter in A..Z
do cat <<- EOF > batch_$letter.sh
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
for AF in $BASE/$letter*.txt_step3; do
Rscript $R_funcP_tools.R
--ptool $R_func/P_tools_linux
--group $AF
--treat $TREAT/pop_exclude24dup
--out $OUTPUT/OUT_$(basename $AF%%_txt_step3)_noregress
--binary-target F; done
EOF
done
This is the error message
basename: missing operand Try 'basename --help' for more information.
bash scripting variable basename
I am using EOF to generate bash scripts that run the Rscripts. In the Rscript I used basename
to specify the output file name.
When I use EOF
to generate a list of bash scripts, I could not get basename
to work. The error message is shown below. I was still able to get the bash scripts generated but the $AF
turned into a blank in both places where it presented. Very strange!
I had the bash script tested and it is working so I know the problem is somewhere between EOF
and basename
.
How can I use basename
with EOF
? Or is there any alternative methods? Thank you.
for letter in A..Z
do cat <<- EOF > batch_$letter.sh
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
for AF in $BASE/$letter*.txt_step3; do
Rscript $R_funcP_tools.R
--ptool $R_func/P_tools_linux
--group $AF
--treat $TREAT/pop_exclude24dup
--out $OUTPUT/OUT_$(basename $AF%%_txt_step3)_noregress
--binary-target F; done
EOF
done
This is the error message
basename: missing operand Try 'basename --help' for more information.
bash scripting variable basename
bash scripting variable basename
edited Mar 27 at 15:29
Molly_K
asked Mar 26 at 21:19
Molly_KMolly_K
275
275
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
Command substitutions like your $(basename ...)
and variables are also expanded in here-documents if the delimiter is not quoted. You should escape the $
from $(basename ...)
and also any $
inside it.
Corrected version of your script:
for letter in A..Z
do cat <<- EOF > batch_$letter.sh
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
for letter in A..Z do
for AF in $BASE/$letter*.txt_step3; do
Rscript $R_funcP_tools.R
--ptool $R_func/P_tools_linux
--group $AF
--treat $TREAT/pop_exclude24dup
--out $OUTPUT/OUT_$(basename $AF%%_txt_step3)_noregress
--binary-target F; done
EOF
done
That is actually indented by tabs; this stupid web interface is turning tabs into spaces, which will probably break <<-
which in any POSIX shell is only stripping tabs, not spaces from before the EOF delimiter and the lines from the here-document.
I actually like your previous version better! haha! Thank you @unclebilly, it works very well and your explanation is great.
– Molly_K
Mar 26 at 21:50
It's still wrong, the$
fromR_func
should be quoted, too. Fixed now.
– Uncle Billy
Mar 26 at 21:51
Yes, it was not fully corrected at the time but enough info was given so I learned from you that I needed to add the backsslah to variables. Thank you again. Very helpful!
– Molly_K
Mar 26 at 21:58
add a comment |
The << EOF
...EOF
construct is called a here-document, and you can put whatever string you like as the delimiter, but EOF
is common.
The issue you're facing is that the here-document acts like a double-quoted string,
so the variables and the command substitution in it are expanded when cat
runs, they're not stored as-is in the resulting file. This is probably not what you want, since you set, e.g. R_func
in the batch_x.sh
you're writing, but $R_func
would expand to whatever value R_func
has in the generating script.
You can prevent this, by quoting the here-doc delimiter, i.e. use cat << 'EOF'
instead. However, this prevents expansion of all variables, so you can't have one expanded and the others not without creating the file in parts or using an unquoted delimiter and escaping all but one variable, as in Uncle Billy's answer.
Now, if I understood your idea correctly, you want to create 26 scripts with each letter hard-coded in one of them. The first script you create (batch_A.sh
) then looks something like this:
module load R/3.5.1
R_func="/home/dir/R_func"
....
for AF in $BASE/A*.txt_step3; do
Rscript $R_funcP_tools.R
...
done
Instead, you could create just one script, as below, and the pass the letter to it as a command line argument. The first command line argument is available as "$1"
:
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
letter=$1
for AF in $BASE/$letter*.txt_step3; do
Rscript "$R_funcP_tools.R"
--ptool "$R_func/P_tools_linux"
--group "$AF"
--treat "$TREAT/pop_exclude24dup"
--out "$OUTPUT/OUT_$(basename "$AF%%_txt_step3")_noregress"
--binary-target F;
done
The variable letter
would now be taken from the command line, so you could run batch.sh A
to process the A
files, etc.
Thank you @ilkkachu, great answer and works well!!
– Molly_K
Mar 26 at 21:50
Hi @ilkkachu, I wonder if you could let me know if there's a way to escape the "stored as-is" under thecat << 'EOF'
circumstance. More specifically, I want$letter
in both bash script filename and within the script.
– Molly_K
Mar 27 at 15:14
You mentioned that creating "one" script and passing the letter to it as a command line argument. Could you elaborate more on that? My original thought is to break down files based on alphabets and run in parallel. However, if there's a way to break down the files individually and run scripts in parallel would be even better, I should work on that.
– Molly_K
Mar 27 at 15:15
@Molly_K, I tried to adapt your script to picking$letter
from the command line. For parallel processing, you could write the script so that it processes just one file, and then runxargs -P
or use GNU parallel to run a bunch of them at a time.
– ilkkachu
Mar 27 at 19:14
There's probably a dozen or so Q&A's about running scripts in parallel here on the site, see e.g. unix.stackexchange.com/q/169326/170373 and unix.stackexchange.com/q/211976/170373 and the search unix.stackexchange.com/…
– ilkkachu
Mar 27 at 19:16
|
show 2 more comments
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f508834%2fhere-document-basename-conflicts%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Command substitutions like your $(basename ...)
and variables are also expanded in here-documents if the delimiter is not quoted. You should escape the $
from $(basename ...)
and also any $
inside it.
Corrected version of your script:
for letter in A..Z
do cat <<- EOF > batch_$letter.sh
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
for letter in A..Z do
for AF in $BASE/$letter*.txt_step3; do
Rscript $R_funcP_tools.R
--ptool $R_func/P_tools_linux
--group $AF
--treat $TREAT/pop_exclude24dup
--out $OUTPUT/OUT_$(basename $AF%%_txt_step3)_noregress
--binary-target F; done
EOF
done
That is actually indented by tabs; this stupid web interface is turning tabs into spaces, which will probably break <<-
which in any POSIX shell is only stripping tabs, not spaces from before the EOF delimiter and the lines from the here-document.
I actually like your previous version better! haha! Thank you @unclebilly, it works very well and your explanation is great.
– Molly_K
Mar 26 at 21:50
It's still wrong, the$
fromR_func
should be quoted, too. Fixed now.
– Uncle Billy
Mar 26 at 21:51
Yes, it was not fully corrected at the time but enough info was given so I learned from you that I needed to add the backsslah to variables. Thank you again. Very helpful!
– Molly_K
Mar 26 at 21:58
add a comment |
Command substitutions like your $(basename ...)
and variables are also expanded in here-documents if the delimiter is not quoted. You should escape the $
from $(basename ...)
and also any $
inside it.
Corrected version of your script:
for letter in A..Z
do cat <<- EOF > batch_$letter.sh
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
for letter in A..Z do
for AF in $BASE/$letter*.txt_step3; do
Rscript $R_funcP_tools.R
--ptool $R_func/P_tools_linux
--group $AF
--treat $TREAT/pop_exclude24dup
--out $OUTPUT/OUT_$(basename $AF%%_txt_step3)_noregress
--binary-target F; done
EOF
done
That is actually indented by tabs; this stupid web interface is turning tabs into spaces, which will probably break <<-
which in any POSIX shell is only stripping tabs, not spaces from before the EOF delimiter and the lines from the here-document.
I actually like your previous version better! haha! Thank you @unclebilly, it works very well and your explanation is great.
– Molly_K
Mar 26 at 21:50
It's still wrong, the$
fromR_func
should be quoted, too. Fixed now.
– Uncle Billy
Mar 26 at 21:51
Yes, it was not fully corrected at the time but enough info was given so I learned from you that I needed to add the backsslah to variables. Thank you again. Very helpful!
– Molly_K
Mar 26 at 21:58
add a comment |
Command substitutions like your $(basename ...)
and variables are also expanded in here-documents if the delimiter is not quoted. You should escape the $
from $(basename ...)
and also any $
inside it.
Corrected version of your script:
for letter in A..Z
do cat <<- EOF > batch_$letter.sh
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
for letter in A..Z do
for AF in $BASE/$letter*.txt_step3; do
Rscript $R_funcP_tools.R
--ptool $R_func/P_tools_linux
--group $AF
--treat $TREAT/pop_exclude24dup
--out $OUTPUT/OUT_$(basename $AF%%_txt_step3)_noregress
--binary-target F; done
EOF
done
That is actually indented by tabs; this stupid web interface is turning tabs into spaces, which will probably break <<-
which in any POSIX shell is only stripping tabs, not spaces from before the EOF delimiter and the lines from the here-document.
Command substitutions like your $(basename ...)
and variables are also expanded in here-documents if the delimiter is not quoted. You should escape the $
from $(basename ...)
and also any $
inside it.
Corrected version of your script:
for letter in A..Z
do cat <<- EOF > batch_$letter.sh
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
for letter in A..Z do
for AF in $BASE/$letter*.txt_step3; do
Rscript $R_funcP_tools.R
--ptool $R_func/P_tools_linux
--group $AF
--treat $TREAT/pop_exclude24dup
--out $OUTPUT/OUT_$(basename $AF%%_txt_step3)_noregress
--binary-target F; done
EOF
done
That is actually indented by tabs; this stupid web interface is turning tabs into spaces, which will probably break <<-
which in any POSIX shell is only stripping tabs, not spaces from before the EOF delimiter and the lines from the here-document.
edited Mar 28 at 5:23
answered Mar 26 at 21:30
Uncle BillyUncle Billy
9358
9358
I actually like your previous version better! haha! Thank you @unclebilly, it works very well and your explanation is great.
– Molly_K
Mar 26 at 21:50
It's still wrong, the$
fromR_func
should be quoted, too. Fixed now.
– Uncle Billy
Mar 26 at 21:51
Yes, it was not fully corrected at the time but enough info was given so I learned from you that I needed to add the backsslah to variables. Thank you again. Very helpful!
– Molly_K
Mar 26 at 21:58
add a comment |
I actually like your previous version better! haha! Thank you @unclebilly, it works very well and your explanation is great.
– Molly_K
Mar 26 at 21:50
It's still wrong, the$
fromR_func
should be quoted, too. Fixed now.
– Uncle Billy
Mar 26 at 21:51
Yes, it was not fully corrected at the time but enough info was given so I learned from you that I needed to add the backsslah to variables. Thank you again. Very helpful!
– Molly_K
Mar 26 at 21:58
I actually like your previous version better! haha! Thank you @unclebilly, it works very well and your explanation is great.
– Molly_K
Mar 26 at 21:50
I actually like your previous version better! haha! Thank you @unclebilly, it works very well and your explanation is great.
– Molly_K
Mar 26 at 21:50
It's still wrong, the
$
from R_func
should be quoted, too. Fixed now.– Uncle Billy
Mar 26 at 21:51
It's still wrong, the
$
from R_func
should be quoted, too. Fixed now.– Uncle Billy
Mar 26 at 21:51
Yes, it was not fully corrected at the time but enough info was given so I learned from you that I needed to add the backsslah to variables. Thank you again. Very helpful!
– Molly_K
Mar 26 at 21:58
Yes, it was not fully corrected at the time but enough info was given so I learned from you that I needed to add the backsslah to variables. Thank you again. Very helpful!
– Molly_K
Mar 26 at 21:58
add a comment |
The << EOF
...EOF
construct is called a here-document, and you can put whatever string you like as the delimiter, but EOF
is common.
The issue you're facing is that the here-document acts like a double-quoted string,
so the variables and the command substitution in it are expanded when cat
runs, they're not stored as-is in the resulting file. This is probably not what you want, since you set, e.g. R_func
in the batch_x.sh
you're writing, but $R_func
would expand to whatever value R_func
has in the generating script.
You can prevent this, by quoting the here-doc delimiter, i.e. use cat << 'EOF'
instead. However, this prevents expansion of all variables, so you can't have one expanded and the others not without creating the file in parts or using an unquoted delimiter and escaping all but one variable, as in Uncle Billy's answer.
Now, if I understood your idea correctly, you want to create 26 scripts with each letter hard-coded in one of them. The first script you create (batch_A.sh
) then looks something like this:
module load R/3.5.1
R_func="/home/dir/R_func"
....
for AF in $BASE/A*.txt_step3; do
Rscript $R_funcP_tools.R
...
done
Instead, you could create just one script, as below, and the pass the letter to it as a command line argument. The first command line argument is available as "$1"
:
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
letter=$1
for AF in $BASE/$letter*.txt_step3; do
Rscript "$R_funcP_tools.R"
--ptool "$R_func/P_tools_linux"
--group "$AF"
--treat "$TREAT/pop_exclude24dup"
--out "$OUTPUT/OUT_$(basename "$AF%%_txt_step3")_noregress"
--binary-target F;
done
The variable letter
would now be taken from the command line, so you could run batch.sh A
to process the A
files, etc.
Thank you @ilkkachu, great answer and works well!!
– Molly_K
Mar 26 at 21:50
Hi @ilkkachu, I wonder if you could let me know if there's a way to escape the "stored as-is" under thecat << 'EOF'
circumstance. More specifically, I want$letter
in both bash script filename and within the script.
– Molly_K
Mar 27 at 15:14
You mentioned that creating "one" script and passing the letter to it as a command line argument. Could you elaborate more on that? My original thought is to break down files based on alphabets and run in parallel. However, if there's a way to break down the files individually and run scripts in parallel would be even better, I should work on that.
– Molly_K
Mar 27 at 15:15
@Molly_K, I tried to adapt your script to picking$letter
from the command line. For parallel processing, you could write the script so that it processes just one file, and then runxargs -P
or use GNU parallel to run a bunch of them at a time.
– ilkkachu
Mar 27 at 19:14
There's probably a dozen or so Q&A's about running scripts in parallel here on the site, see e.g. unix.stackexchange.com/q/169326/170373 and unix.stackexchange.com/q/211976/170373 and the search unix.stackexchange.com/…
– ilkkachu
Mar 27 at 19:16
|
show 2 more comments
The << EOF
...EOF
construct is called a here-document, and you can put whatever string you like as the delimiter, but EOF
is common.
The issue you're facing is that the here-document acts like a double-quoted string,
so the variables and the command substitution in it are expanded when cat
runs, they're not stored as-is in the resulting file. This is probably not what you want, since you set, e.g. R_func
in the batch_x.sh
you're writing, but $R_func
would expand to whatever value R_func
has in the generating script.
You can prevent this, by quoting the here-doc delimiter, i.e. use cat << 'EOF'
instead. However, this prevents expansion of all variables, so you can't have one expanded and the others not without creating the file in parts or using an unquoted delimiter and escaping all but one variable, as in Uncle Billy's answer.
Now, if I understood your idea correctly, you want to create 26 scripts with each letter hard-coded in one of them. The first script you create (batch_A.sh
) then looks something like this:
module load R/3.5.1
R_func="/home/dir/R_func"
....
for AF in $BASE/A*.txt_step3; do
Rscript $R_funcP_tools.R
...
done
Instead, you could create just one script, as below, and the pass the letter to it as a command line argument. The first command line argument is available as "$1"
:
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
letter=$1
for AF in $BASE/$letter*.txt_step3; do
Rscript "$R_funcP_tools.R"
--ptool "$R_func/P_tools_linux"
--group "$AF"
--treat "$TREAT/pop_exclude24dup"
--out "$OUTPUT/OUT_$(basename "$AF%%_txt_step3")_noregress"
--binary-target F;
done
The variable letter
would now be taken from the command line, so you could run batch.sh A
to process the A
files, etc.
Thank you @ilkkachu, great answer and works well!!
– Molly_K
Mar 26 at 21:50
Hi @ilkkachu, I wonder if you could let me know if there's a way to escape the "stored as-is" under thecat << 'EOF'
circumstance. More specifically, I want$letter
in both bash script filename and within the script.
– Molly_K
Mar 27 at 15:14
You mentioned that creating "one" script and passing the letter to it as a command line argument. Could you elaborate more on that? My original thought is to break down files based on alphabets and run in parallel. However, if there's a way to break down the files individually and run scripts in parallel would be even better, I should work on that.
– Molly_K
Mar 27 at 15:15
@Molly_K, I tried to adapt your script to picking$letter
from the command line. For parallel processing, you could write the script so that it processes just one file, and then runxargs -P
or use GNU parallel to run a bunch of them at a time.
– ilkkachu
Mar 27 at 19:14
There's probably a dozen or so Q&A's about running scripts in parallel here on the site, see e.g. unix.stackexchange.com/q/169326/170373 and unix.stackexchange.com/q/211976/170373 and the search unix.stackexchange.com/…
– ilkkachu
Mar 27 at 19:16
|
show 2 more comments
The << EOF
...EOF
construct is called a here-document, and you can put whatever string you like as the delimiter, but EOF
is common.
The issue you're facing is that the here-document acts like a double-quoted string,
so the variables and the command substitution in it are expanded when cat
runs, they're not stored as-is in the resulting file. This is probably not what you want, since you set, e.g. R_func
in the batch_x.sh
you're writing, but $R_func
would expand to whatever value R_func
has in the generating script.
You can prevent this, by quoting the here-doc delimiter, i.e. use cat << 'EOF'
instead. However, this prevents expansion of all variables, so you can't have one expanded and the others not without creating the file in parts or using an unquoted delimiter and escaping all but one variable, as in Uncle Billy's answer.
Now, if I understood your idea correctly, you want to create 26 scripts with each letter hard-coded in one of them. The first script you create (batch_A.sh
) then looks something like this:
module load R/3.5.1
R_func="/home/dir/R_func"
....
for AF in $BASE/A*.txt_step3; do
Rscript $R_funcP_tools.R
...
done
Instead, you could create just one script, as below, and the pass the letter to it as a command line argument. The first command line argument is available as "$1"
:
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
letter=$1
for AF in $BASE/$letter*.txt_step3; do
Rscript "$R_funcP_tools.R"
--ptool "$R_func/P_tools_linux"
--group "$AF"
--treat "$TREAT/pop_exclude24dup"
--out "$OUTPUT/OUT_$(basename "$AF%%_txt_step3")_noregress"
--binary-target F;
done
The variable letter
would now be taken from the command line, so you could run batch.sh A
to process the A
files, etc.
The << EOF
...EOF
construct is called a here-document, and you can put whatever string you like as the delimiter, but EOF
is common.
The issue you're facing is that the here-document acts like a double-quoted string,
so the variables and the command substitution in it are expanded when cat
runs, they're not stored as-is in the resulting file. This is probably not what you want, since you set, e.g. R_func
in the batch_x.sh
you're writing, but $R_func
would expand to whatever value R_func
has in the generating script.
You can prevent this, by quoting the here-doc delimiter, i.e. use cat << 'EOF'
instead. However, this prevents expansion of all variables, so you can't have one expanded and the others not without creating the file in parts or using an unquoted delimiter and escaping all but one variable, as in Uncle Billy's answer.
Now, if I understood your idea correctly, you want to create 26 scripts with each letter hard-coded in one of them. The first script you create (batch_A.sh
) then looks something like this:
module load R/3.5.1
R_func="/home/dir/R_func"
....
for AF in $BASE/A*.txt_step3; do
Rscript $R_funcP_tools.R
...
done
Instead, you could create just one script, as below, and the pass the letter to it as a command line argument. The first command line argument is available as "$1"
:
#!/bin/bash
module load R/3.5.1
R_func="/home/dir/R_func"
TREAT="/home/dir/POP"
BASE="/home/dir/base"
OUTPUT="/home/dir/tmp"
letter=$1
for AF in $BASE/$letter*.txt_step3; do
Rscript "$R_funcP_tools.R"
--ptool "$R_func/P_tools_linux"
--group "$AF"
--treat "$TREAT/pop_exclude24dup"
--out "$OUTPUT/OUT_$(basename "$AF%%_txt_step3")_noregress"
--binary-target F;
done
The variable letter
would now be taken from the command line, so you could run batch.sh A
to process the A
files, etc.
edited Mar 27 at 19:09
answered Mar 26 at 21:35
ilkkachuilkkachu
63.2k10104181
63.2k10104181
Thank you @ilkkachu, great answer and works well!!
– Molly_K
Mar 26 at 21:50
Hi @ilkkachu, I wonder if you could let me know if there's a way to escape the "stored as-is" under thecat << 'EOF'
circumstance. More specifically, I want$letter
in both bash script filename and within the script.
– Molly_K
Mar 27 at 15:14
You mentioned that creating "one" script and passing the letter to it as a command line argument. Could you elaborate more on that? My original thought is to break down files based on alphabets and run in parallel. However, if there's a way to break down the files individually and run scripts in parallel would be even better, I should work on that.
– Molly_K
Mar 27 at 15:15
@Molly_K, I tried to adapt your script to picking$letter
from the command line. For parallel processing, you could write the script so that it processes just one file, and then runxargs -P
or use GNU parallel to run a bunch of them at a time.
– ilkkachu
Mar 27 at 19:14
There's probably a dozen or so Q&A's about running scripts in parallel here on the site, see e.g. unix.stackexchange.com/q/169326/170373 and unix.stackexchange.com/q/211976/170373 and the search unix.stackexchange.com/…
– ilkkachu
Mar 27 at 19:16
|
show 2 more comments
Thank you @ilkkachu, great answer and works well!!
– Molly_K
Mar 26 at 21:50
Hi @ilkkachu, I wonder if you could let me know if there's a way to escape the "stored as-is" under thecat << 'EOF'
circumstance. More specifically, I want$letter
in both bash script filename and within the script.
– Molly_K
Mar 27 at 15:14
You mentioned that creating "one" script and passing the letter to it as a command line argument. Could you elaborate more on that? My original thought is to break down files based on alphabets and run in parallel. However, if there's a way to break down the files individually and run scripts in parallel would be even better, I should work on that.
– Molly_K
Mar 27 at 15:15
@Molly_K, I tried to adapt your script to picking$letter
from the command line. For parallel processing, you could write the script so that it processes just one file, and then runxargs -P
or use GNU parallel to run a bunch of them at a time.
– ilkkachu
Mar 27 at 19:14
There's probably a dozen or so Q&A's about running scripts in parallel here on the site, see e.g. unix.stackexchange.com/q/169326/170373 and unix.stackexchange.com/q/211976/170373 and the search unix.stackexchange.com/…
– ilkkachu
Mar 27 at 19:16
Thank you @ilkkachu, great answer and works well!!
– Molly_K
Mar 26 at 21:50
Thank you @ilkkachu, great answer and works well!!
– Molly_K
Mar 26 at 21:50
Hi @ilkkachu, I wonder if you could let me know if there's a way to escape the "stored as-is" under the
cat << 'EOF'
circumstance. More specifically, I want $letter
in both bash script filename and within the script.– Molly_K
Mar 27 at 15:14
Hi @ilkkachu, I wonder if you could let me know if there's a way to escape the "stored as-is" under the
cat << 'EOF'
circumstance. More specifically, I want $letter
in both bash script filename and within the script.– Molly_K
Mar 27 at 15:14
You mentioned that creating "one" script and passing the letter to it as a command line argument. Could you elaborate more on that? My original thought is to break down files based on alphabets and run in parallel. However, if there's a way to break down the files individually and run scripts in parallel would be even better, I should work on that.
– Molly_K
Mar 27 at 15:15
You mentioned that creating "one" script and passing the letter to it as a command line argument. Could you elaborate more on that? My original thought is to break down files based on alphabets and run in parallel. However, if there's a way to break down the files individually and run scripts in parallel would be even better, I should work on that.
– Molly_K
Mar 27 at 15:15
@Molly_K, I tried to adapt your script to picking
$letter
from the command line. For parallel processing, you could write the script so that it processes just one file, and then run xargs -P
or use GNU parallel to run a bunch of them at a time.– ilkkachu
Mar 27 at 19:14
@Molly_K, I tried to adapt your script to picking
$letter
from the command line. For parallel processing, you could write the script so that it processes just one file, and then run xargs -P
or use GNU parallel to run a bunch of them at a time.– ilkkachu
Mar 27 at 19:14
There's probably a dozen or so Q&A's about running scripts in parallel here on the site, see e.g. unix.stackexchange.com/q/169326/170373 and unix.stackexchange.com/q/211976/170373 and the search unix.stackexchange.com/…
– ilkkachu
Mar 27 at 19:16
There's probably a dozen or so Q&A's about running scripts in parallel here on the site, see e.g. unix.stackexchange.com/q/169326/170373 and unix.stackexchange.com/q/211976/170373 and the search unix.stackexchange.com/…
– ilkkachu
Mar 27 at 19:16
|
show 2 more comments
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f508834%2fhere-document-basename-conflicts%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
-basename, bash, scripting, variable