is 'sed' thread safe2019 Community Moderator ElectionIn-place editing using sed on AIXWhat should someone know about using Python scripts in the shell?Nexenta bash script uses /usr/sun/bin/sed instead of /usr/bin/sedLeft and right square brackets treated differently by sed/bashsystemd daemon & python getting the wrong timesed multiple statements within a single command not workingHow to use sed to substitute strings which has “” in it?Python process can't create a file in a directory, keeps getting `permission denied` IOErrorIs a light weight process attached to a kernel thread in Linux?Extract every 2 lines from a 40 lines file and create a new fileRHEL upload / pull scripts from people without access to the server itseld
How strictly should I take "Candidates must be local"?
Can a bounded number sequence be strictly ascending?
How to create a hard link to an inode (ext4)?
Should QA ask requirements to developers?
What does C-53 signify?
Can Mathematica be used to create an Artistic 3D extrusion from a 2D image and wrap a line pattern around it?
How do I express some one as a black person?
Why does the negative sign arise in this thermodynamic relation?
Why was Goose renamed from Chewie for the Captain Marvel film?
Best approach to update all entries in a list that is paginated?
Why would one plane in this picture not have gear down yet?
Word-Letter Ladder
If one operation is wrong then the whole transaction is wrong
Should I tell my boss the work he did was worthless
Intuition behind counterexample of Euler's sum of powers conjecture
Unreachable code, but reachable with exception
Why is there a voltage between the mains ground and my radiator?
Do items de-spawn in Diablo?
How did the power source of Mar-Vell's aircraft end up with her?
Peter's Strange Word
How can The Temple of Elementary Evil reliably protect itself against kinetic bombardment?
Should I take out a loan for a friend to invest on my behalf?
Built-In Shelves/Bookcases - IKEA vs Built
Grey hair or white hair
is 'sed' thread safe
2019 Community Moderator ElectionIn-place editing using sed on AIXWhat should someone know about using Python scripts in the shell?Nexenta bash script uses /usr/sun/bin/sed instead of /usr/bin/sedLeft and right square brackets treated differently by sed/bashsystemd daemon & python getting the wrong timesed multiple statements within a single command not workingHow to use sed to substitute strings which has “” in it?Python process can't create a file in a directory, keeps getting `permission denied` IOErrorIs a light weight process attached to a kernel thread in Linux?Extract every 2 lines from a 40 lines file and create a new fileRHEL upload / pull scripts from people without access to the server itseld
If I have a shell/python script that uses sed to modify a file in place based on user inputs, and then two users run the same script at the same time or approx. same time, is 'sed' thread safe ? Or perhaps it is not an issue because the file_descripor that was opened by the first thread will be used to lock the file anyway ? thx
linux sed python
edited Mar 6 at 21:52
Jeff Schaller
43.3k1159139
asked Mar 6 at 21:34
terreysterreys
2113
add a comment |
If I have a shell/python script that uses sed to modify a file in place based on user inputs, and then two users run the same script at the same time or approx. same time, is 'sed' thread safe ? Or perhaps it is not an issue because the file_descripor that was opened by the first thread will be used to lock the file anyway ? thx
linux sed python
edited Mar 6 at 21:52
Jeff Schaller
43.3k1159139
asked Mar 6 at 21:34
terreysterreys
2113
2
Slightly more info wanted. How do you usesedfrom Python (and why, can't Python do things like that fairly effortlessly?).
– Kusalananda
Mar 6 at 21:38
add a comment |
If I have a shell/python script that uses sed to modify a file in place based on user inputs, and then two users run the same script at the same time or approx. same time, is 'sed' thread safe ? Or perhaps it is not an issue because the file_descripor that was opened by the first thread will be used to lock the file anyway ? thx
linux sed python
edited Mar 6 at 21:52
Jeff Schaller
43.3k1159139
asked Mar 6 at 21:34
terreysterreys
2113
If I have a shell/python script that uses sed to modify a file in place based on user inputs, and then two users run the same script at the same time or approx. same time, is 'sed' thread safe ? Or perhaps it is not an issue because the file_descripor that was opened by the first thread will be used to lock the file anyway ? thx
linux sed python
linux sed python
edited Mar 6 at 21:52
Jeff Schaller
43.3k1159139
asked Mar 6 at 21:34
terreysterreys
2113
edited Mar 6 at 21:52
Jeff Schaller
43.3k1159139
asked Mar 6 at 21:34
terreysterreys
2113
edited Mar 6 at 21:52
Jeff Schaller
43.3k1159139
edited Mar 6 at 21:52
Jeff Schaller
43.3k1159139
edited Mar 6 at 21:52
Jeff Schaller
43.3k1159139
43.3k1159139
asked Mar 6 at 21:34
terreysterreys
2113
asked Mar 6 at 21:34
terreysterreys
2113
asked Mar 6 at 21:34
terreysterreys
2113
2113
2
Slightly more info wanted. How do you usesedfrom Python (and why, can't Python do things like that fairly effortlessly?).
– Kusalananda
Mar 6 at 21:38
add a comment |
2
Slightly more info wanted. How do you usesedfrom Python (and why, can't Python do things like that fairly effortlessly?).
– Kusalananda
Mar 6 at 21:38
2
2
Slightly more info wanted. How do you use
sed from Python (and why, can't Python do things like that fairly effortlessly?).– Kusalananda
Mar 6 at 21:38
Slightly more info wanted. How do you use
sed from Python (and why, can't Python do things like that fairly effortlessly?).– Kusalananda
Mar 6 at 21:38
add a comment |
1 Answer
1
active
oldest
votes
I'm not going to nitpick on the awful terminology, but yes, GNU sed with its -i ("in-place") flag could be safely used by more than one process at the same time without any extra locking, because sed is not actually modifying the file in-place, but it's redirecting the output to a temporary file, and if everything goes well, it will rename(2) (move) the temporary file to the original file, and the rename(2) is guaranteed to be atomic:
$ strace sed -i s/o/e/g foo.txt
open("foo.txt", O_RDONLY) = 3
...
open("./sedDe80VL", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
...
read(3, "foon", 4096) = 4
...
write(4, "feen", 4) = 4
read(3, "", 4096) = 0
...
close(3) = 0
close(4) = 0
rename("./sedDe80VL", "foo.txt") = 0
At any point, foo.txt will refer either to the complete original file or to the complete processed file, never to something in between the two.
Notes:
This does not handle the case where more than one process starts editing a file without waiting for the other processes to have finished editing it, in which case only the process which finishes last "wins" (ie wipes the changes performed by the other processes). This is not a matter of data integrity, and cannot be handled without higher level coordination between the processes (blindly locking the file will lead to deadlocks).
Currently, GNU sed will copy the standard file permissions into the new inode, but not the ACLs and extended attributes. If using sed -i on such a file, all that extra metadata will be lost. IMHO that's more of a feature than a bug or limitation.
perl -i used to work very differently from sed -i until version 5.28; it used to first make a temporary copy of the file, truncate to the original file, and redirect the output to it. That was preserving the original inode number and extra metadata, but would completely trash the content of the file in the case where the perl -i process was interrupted or more than one perl -i process was editing the file at the same time. See the discussion, the original commit (which was subsequently improved) and the changelog in perl5280delta.
answered Mar 6 at 21:47
mosvymosvy
8,1921632
3
But what can happen is that two copies ofsedstart reading the file, make different changes, storing them to their respective temporary files, which are then renamed into place one after other, without regard for the fact that there was anothersedworking on the file at the same time. The changes made by one of thesedprocesses would be lost.
– ilkkachu
Mar 6 at 22:05
2
@ilkkachu that will still be completely consistent -- the file will be either modified by both processes in turn (in any order) or by just one of them. At no point will the file contain garbage resulting from both processes modifying it at the same time.
– mosvy
Mar 6 at 22:15
8
it won't be total garbage, but having changes lost can still be an issue. The question refers to the file "being locked", and locking would usually refer to the secondsedprocess waiting until the first completed. That might just be awful terminology on their part; it's hard to say what they really care about, but that particular issue is still possible.
– ilkkachu
Mar 6 at 22:25
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504810%2fis-sed-thread-safe%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I'm not going to nitpick on the awful terminology, but yes, GNU sed with its -i ("in-place") flag could be safely used by more than one process at the same time without any extra locking, because sed is not actually modifying the file in-place, but it's redirecting the output to a temporary file, and if everything goes well, it will rename(2) (move) the temporary file to the original file, and the rename(2) is guaranteed to be atomic:
$ strace sed -i s/o/e/g foo.txt
open("foo.txt", O_RDONLY) = 3
...
open("./sedDe80VL", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
...
read(3, "foon", 4096) = 4
...
write(4, "feen", 4) = 4
read(3, "", 4096) = 0
...
close(3) = 0
close(4) = 0
rename("./sedDe80VL", "foo.txt") = 0
At any point, foo.txt will refer either to the complete original file or to the complete processed file, never to something in between the two.
Notes:
This does not handle the case where more than one process starts editing a file without waiting for the other processes to have finished editing it, in which case only the process which finishes last "wins" (ie wipes the changes performed by the other processes). This is not a matter of data integrity, and cannot be handled without higher level coordination between the processes (blindly locking the file will lead to deadlocks).
Currently, GNU sed will copy the standard file permissions into the new inode, but not the ACLs and extended attributes. If using sed -i on such a file, all that extra metadata will be lost. IMHO that's more of a feature than a bug or limitation.
perl -i used to work very differently from sed -i until version 5.28; it used to first make a temporary copy of the file, truncate to the original file, and redirect the output to it. That was preserving the original inode number and extra metadata, but would completely trash the content of the file in the case where the perl -i process was interrupted or more than one perl -i process was editing the file at the same time. See the discussion, the original commit (which was subsequently improved) and the changelog in perl5280delta.
answered Mar 6 at 21:47
mosvymosvy
8,1921632
3
But what can happen is that two copies ofsedstart reading the file, make different changes, storing them to their respective temporary files, which are then renamed into place one after other, without regard for the fact that there was anothersedworking on the file at the same time. The changes made by one of thesedprocesses would be lost.
– ilkkachu
Mar 6 at 22:05
2
@ilkkachu that will still be completely consistent -- the file will be either modified by both processes in turn (in any order) or by just one of them. At no point will the file contain garbage resulting from both processes modifying it at the same time.
– mosvy
Mar 6 at 22:15
8
it won't be total garbage, but having changes lost can still be an issue. The question refers to the file "being locked", and locking would usually refer to the secondsedprocess waiting until the first completed. That might just be awful terminology on their part; it's hard to say what they really care about, but that particular issue is still possible.
– ilkkachu
Mar 6 at 22:25
add a comment |
I'm not going to nitpick on the awful terminology, but yes, GNU sed with its -i ("in-place") flag could be safely used by more than one process at the same time without any extra locking, because sed is not actually modifying the file in-place, but it's redirecting the output to a temporary file, and if everything goes well, it will rename(2) (move) the temporary file to the original file, and the rename(2) is guaranteed to be atomic:
$ strace sed -i s/o/e/g foo.txt
open("foo.txt", O_RDONLY) = 3
...
open("./sedDe80VL", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
...
read(3, "foon", 4096) = 4
...
write(4, "feen", 4) = 4
read(3, "", 4096) = 0
...
close(3) = 0
close(4) = 0
rename("./sedDe80VL", "foo.txt") = 0
At any point, foo.txt will refer either to the complete original file or to the complete processed file, never to something in between the two.
Notes:
This does not handle the case where more than one process starts editing a file without waiting for the other processes to have finished editing it, in which case only the process which finishes last "wins" (ie wipes the changes performed by the other processes). This is not a matter of data integrity, and cannot be handled without higher level coordination between the processes (blindly locking the file will lead to deadlocks).
Currently, GNU sed will copy the standard file permissions into the new inode, but not the ACLs and extended attributes. If using sed -i on such a file, all that extra metadata will be lost. IMHO that's more of a feature than a bug or limitation.
perl -i used to work very differently from sed -i until version 5.28; it used to first make a temporary copy of the file, truncate to the original file, and redirect the output to it. That was preserving the original inode number and extra metadata, but would completely trash the content of the file in the case where the perl -i process was interrupted or more than one perl -i process was editing the file at the same time. See the discussion, the original commit (which was subsequently improved) and the changelog in perl5280delta.
answered Mar 6 at 21:47
mosvymosvy
8,1921632
3
But what can happen is that two copies ofsedstart reading the file, make different changes, storing them to their respective temporary files, which are then renamed into place one after other, without regard for the fact that there was anothersedworking on the file at the same time. The changes made by one of thesedprocesses would be lost.
– ilkkachu
Mar 6 at 22:05
2
@ilkkachu that will still be completely consistent -- the file will be either modified by both processes in turn (in any order) or by just one of them. At no point will the file contain garbage resulting from both processes modifying it at the same time.
– mosvy
Mar 6 at 22:15
8
it won't be total garbage, but having changes lost can still be an issue. The question refers to the file "being locked", and locking would usually refer to the secondsedprocess waiting until the first completed. That might just be awful terminology on their part; it's hard to say what they really care about, but that particular issue is still possible.
– ilkkachu
Mar 6 at 22:25
add a comment |
I'm not going to nitpick on the awful terminology, but yes, GNU sed with its -i ("in-place") flag could be safely used by more than one process at the same time without any extra locking, because sed is not actually modifying the file in-place, but it's redirecting the output to a temporary file, and if everything goes well, it will rename(2) (move) the temporary file to the original file, and the rename(2) is guaranteed to be atomic:
$ strace sed -i s/o/e/g foo.txt
open("foo.txt", O_RDONLY) = 3
...
open("./sedDe80VL", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
...
read(3, "foon", 4096) = 4
...
write(4, "feen", 4) = 4
read(3, "", 4096) = 0
...
close(3) = 0
close(4) = 0
rename("./sedDe80VL", "foo.txt") = 0
At any point, foo.txt will refer either to the complete original file or to the complete processed file, never to something in between the two.
Notes:
This does not handle the case where more than one process starts editing a file without waiting for the other processes to have finished editing it, in which case only the process which finishes last "wins" (ie wipes the changes performed by the other processes). This is not a matter of data integrity, and cannot be handled without higher level coordination between the processes (blindly locking the file will lead to deadlocks).
Currently, GNU sed will copy the standard file permissions into the new inode, but not the ACLs and extended attributes. If using sed -i on such a file, all that extra metadata will be lost. IMHO that's more of a feature than a bug or limitation.
perl -i used to work very differently from sed -i until version 5.28; it used to first make a temporary copy of the file, truncate to the original file, and redirect the output to it. That was preserving the original inode number and extra metadata, but would completely trash the content of the file in the case where the perl -i process was interrupted or more than one perl -i process was editing the file at the same time. See the discussion, the original commit (which was subsequently improved) and the changelog in perl5280delta.
answered Mar 6 at 21:47
mosvymosvy
8,1921632
I'm not going to nitpick on the awful terminology, but yes, GNU sed with its -i ("in-place") flag could be safely used by more than one process at the same time without any extra locking, because sed is not actually modifying the file in-place, but it's redirecting the output to a temporary file, and if everything goes well, it will rename(2) (move) the temporary file to the original file, and the rename(2) is guaranteed to be atomic:
$ strace sed -i s/o/e/g foo.txt
open("foo.txt", O_RDONLY) = 3
...
open("./sedDe80VL", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
...
read(3, "foon", 4096) = 4
...
write(4, "feen", 4) = 4
read(3, "", 4096) = 0
...
close(3) = 0
close(4) = 0
rename("./sedDe80VL", "foo.txt") = 0
At any point, foo.txt will refer either to the complete original file or to the complete processed file, never to something in between the two.
Notes:
This does not handle the case where more than one process starts editing a file without waiting for the other processes to have finished editing it, in which case only the process which finishes last "wins" (ie wipes the changes performed by the other processes). This is not a matter of data integrity, and cannot be handled without higher level coordination between the processes (blindly locking the file will lead to deadlocks).
Currently, GNU sed will copy the standard file permissions into the new inode, but not the ACLs and extended attributes. If using sed -i on such a file, all that extra metadata will be lost. IMHO that's more of a feature than a bug or limitation.
perl -i used to work very differently from sed -i until version 5.28; it used to first make a temporary copy of the file, truncate to the original file, and redirect the output to it. That was preserving the original inode number and extra metadata, but would completely trash the content of the file in the case where the perl -i process was interrupted or more than one perl -i process was editing the file at the same time. See the discussion, the original commit (which was subsequently improved) and the changelog in perl5280delta.
answered Mar 6 at 21:47
mosvymosvy
8,1921632
edited 2 hours ago
answered Mar 6 at 21:47
mosvymosvy
8,1921632
answered Mar 6 at 21:47
mosvymosvy
8,1921632
answered Mar 6 at 21:47
mosvymosvy
8,1921632
8,1921632
3
But what can happen is that two copies ofsedstart reading the file, make different changes, storing them to their respective temporary files, which are then renamed into place one after other, without regard for the fact that there was anothersedworking on the file at the same time. The changes made by one of thesedprocesses would be lost.
– ilkkachu
Mar 6 at 22:05
2
@ilkkachu that will still be completely consistent -- the file will be either modified by both processes in turn (in any order) or by just one of them. At no point will the file contain garbage resulting from both processes modifying it at the same time.
– mosvy
Mar 6 at 22:15
8
it won't be total garbage, but having changes lost can still be an issue. The question refers to the file "being locked", and locking would usually refer to the secondsedprocess waiting until the first completed. That might just be awful terminology on their part; it's hard to say what they really care about, but that particular issue is still possible.
– ilkkachu
Mar 6 at 22:25
add a comment |
3
But what can happen is that two copies ofsedstart reading the file, make different changes, storing them to their respective temporary files, which are then renamed into place one after other, without regard for the fact that there was anothersedworking on the file at the same time. The changes made by one of thesedprocesses would be lost.
– ilkkachu
Mar 6 at 22:05
2
@ilkkachu that will still be completely consistent -- the file will be either modified by both processes in turn (in any order) or by just one of them. At no point will the file contain garbage resulting from both processes modifying it at the same time.
– mosvy
Mar 6 at 22:15
8
it won't be total garbage, but having changes lost can still be an issue. The question refers to the file "being locked", and locking would usually refer to the secondsedprocess waiting until the first completed. That might just be awful terminology on their part; it's hard to say what they really care about, but that particular issue is still possible.
– ilkkachu
Mar 6 at 22:25
3
3
But what can happen is that two copies of
sed start reading the file, make different changes, storing them to their respective temporary files, which are then renamed into place one after other, without regard for the fact that there was another sed working on the file at the same time. The changes made by one of the sed processes would be lost.– ilkkachu
Mar 6 at 22:05
But what can happen is that two copies of
sed start reading the file, make different changes, storing them to their respective temporary files, which are then renamed into place one after other, without regard for the fact that there was another sed working on the file at the same time. The changes made by one of the sed processes would be lost.– ilkkachu
Mar 6 at 22:05
2
2
@ilkkachu that will still be completely consistent -- the file will be either modified by both processes in turn (in any order) or by just one of them. At no point will the file contain garbage resulting from both processes modifying it at the same time.
– mosvy
Mar 6 at 22:15
@ilkkachu that will still be completely consistent -- the file will be either modified by both processes in turn (in any order) or by just one of them. At no point will the file contain garbage resulting from both processes modifying it at the same time.
– mosvy
Mar 6 at 22:15
8
8
it won't be total garbage, but having changes lost can still be an issue. The question refers to the file "being locked", and locking would usually refer to the second
sed process waiting until the first completed. That might just be awful terminology on their part; it's hard to say what they really care about, but that particular issue is still possible.– ilkkachu
Mar 6 at 22:25
it won't be total garbage, but having changes lost can still be an issue. The question refers to the file "being locked", and locking would usually refer to the second
sed process waiting until the first completed. That might just be awful terminology on their part; it's hard to say what they really care about, but that particular issue is still possible.– ilkkachu
Mar 6 at 22:25
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504810%2fis-sed-thread-safe%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
-linux, python, sed
2
Slightly more info wanted. How do you use
sedfrom Python (and why, can't Python do things like that fairly effortlessly?).– Kusalananda
Mar 6 at 21:38