Remove duplicates from particular columns The 2019 Stack Overflow Developer Survey Results Are In Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern) 2019 Community Moderator Election ResultsHow does awk '!a[$0]++' work?remove duplicated pattern/entries within each field in CSV fileRemove duplicate fields in a given columnAppend mth and nth columns of a file with the columns of another fileHow to remove the lines that have empty fields?identifying duplicate entries from a file on unix serverNeed to print value from 12th column of a particular row that matches two variables in a CSV fileHow can i print unique value of particular column and then print rest of the columns in next lineRemove specific columns from csv using awkawk - dynamically format tab-separated columnshow to remove comma and strings after a comma in a file?extracting information from a column
Why did Peik Lin say, "I'm not an animal"?
What can I do to 'burn' a journal?
Did the UK government pay "millions and millions of dollars" to try to snag Julian Assange?
How to make Illustrator type tool selection automatically adapt with text length
What to do when moving next to a bird sanctuary with a loosely-domesticated cat?
Keeping a retro style to sci-fi spaceships?
"... to apply for a visa" or "... and applied for a visa"?
What information about me do stores get via my credit card?
Why can't wing-mounted spoilers be used to steepen approaches?
60's-70's movie: home appliances revolting against the owners
ELI5: Why do they say that Israel would have been the fourth country to land a spacecraft on the Moon and why do they call it low cost?
How to politely respond to generic emails requesting a PhD/job in my lab? Without wasting too much time
Could an empire control the whole planet with today's comunication methods?
how can a perfect fourth interval be considered either consonant or dissonant?
Make it rain characters
Is an up-to-date browser secure on an out-of-date OS?
Is 'stolen' appropriate word?
Is this wall load bearing? Blueprints and photos attached
Can we generate random numbers using irrational numbers like π and e?
For what reasons would an animal species NOT cross a *horizontal* land bridge?
Sub-subscripts in strings cause different spacings than subscripts
Do working physicists consider Newtonian mechanics to be "falsified"?
number sequence puzzle deep six
One-dimensional Japanese puzzle
Remove duplicates from particular columns
The 2019 Stack Overflow Developer Survey Results Are In
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)
2019 Community Moderator Election ResultsHow does awk '!a[$0]++' work?remove duplicated pattern/entries within each field in CSV fileRemove duplicate fields in a given columnAppend mth and nth columns of a file with the columns of another fileHow to remove the lines that have empty fields?identifying duplicate entries from a file on unix serverNeed to print value from 12th column of a particular row that matches two variables in a CSV fileHow can i print unique value of particular column and then print rest of the columns in next lineRemove specific columns from csv using awkawk - dynamically format tab-separated columnshow to remove comma and strings after a comma in a file?extracting information from a column
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I have a file in below format, where columns are separated with comma.
[1], Value1, UAC, AB
[2.2], Check1, BOH D2A D2A BOH, SD
[63], name2, MFB MFB, k
...
I want to remove duplicate values from column (say 3rdcolumn) like below:
[1], Value1, UAC, AB
[2.2], Check1, BOH D2A , SD
[63], name2, MFB, k
...
How to use uniq or AWK for particular column.
text-processing awk sed csv
edited yesterday
αғsнιη
17.2k103069
asked yesterday
Jack15Jack15
83
New contributor
Jack15 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
I have a file in below format, where columns are separated with comma.
[1], Value1, UAC, AB
[2.2], Check1, BOH D2A D2A BOH, SD
[63], name2, MFB MFB, k
...
I want to remove duplicate values from column (say 3rdcolumn) like below:
[1], Value1, UAC, AB
[2.2], Check1, BOH D2A , SD
[63], name2, MFB, k
...
How to use uniq or AWK for particular column.
text-processing awk sed csv
edited yesterday
αғsнιη
17.2k103069
asked yesterday
Jack15Jack15
83
New contributor
Jack15 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
order of entries after removing supplicates in 3rd column is matter?
– αғsнιη
yesterday
no matter, But contents should be in same line.
– Jack15
yesterday
add a comment |
I have a file in below format, where columns are separated with comma.
[1], Value1, UAC, AB
[2.2], Check1, BOH D2A D2A BOH, SD
[63], name2, MFB MFB, k
...
I want to remove duplicate values from column (say 3rdcolumn) like below:
[1], Value1, UAC, AB
[2.2], Check1, BOH D2A , SD
[63], name2, MFB, k
...
How to use uniq or AWK for particular column.
text-processing awk sed csv
edited yesterday
αғsнιη
17.2k103069
asked yesterday
Jack15Jack15
83
New contributor
Jack15 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I have a file in below format, where columns are separated with comma.
[1], Value1, UAC, AB
[2.2], Check1, BOH D2A D2A BOH, SD
[63], name2, MFB MFB, k
...
I want to remove duplicate values from column (say 3rdcolumn) like below:
[1], Value1, UAC, AB
[2.2], Check1, BOH D2A , SD
[63], name2, MFB, k
...
How to use uniq or AWK for particular column.
text-processing awk sed csv
text-processing awk sed csv
edited yesterday
αғsнιη
17.2k103069
asked yesterday
Jack15Jack15
83
New contributor
Jack15 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited yesterday
αғsнιη
17.2k103069
asked yesterday
Jack15Jack15
83
New contributor
Jack15 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited yesterday
αғsнιη
17.2k103069
edited yesterday
αғsнιη
17.2k103069
edited yesterday
αғsнιη
17.2k103069
17.2k103069
asked yesterday
Jack15Jack15
83
New contributor
Jack15 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
Jack15Jack15
83
asked yesterday
Jack15Jack15
83
83
New contributor
Jack15 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Jack15 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Jack15 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
order of entries after removing supplicates in 3rd column is matter?
– αғsнιη
yesterday
no matter, But contents should be in same line.
– Jack15
yesterday
add a comment |
order of entries after removing supplicates in 3rd column is matter?
– αғsнιη
yesterday
no matter, But contents should be in same line.
– Jack15
yesterday
order of entries after removing supplicates in 3rd column is matter?
– αғsнιη
yesterday
order of entries after removing supplicates in 3rd column is matter?
– αғsнιη
yesterday
no matter, But contents should be in same line.
– Jack15
yesterday
no matter, But contents should be in same line.
– Jack15
yesterday
add a comment |
2 Answers
2
active
oldest
votes
with awk:
awk -F, '
printf $1 FS $2 FS;
split($3, arr, / +/); for(val in arr) !uniq_arr[ arr[val] ]++;
for (key in uniq_arr)
printf (key!="")? SPACE key:""; SPACE=" "; delete uniq_arr[key]
;
printf FS $4"n"
' infile
[1], Value1, UAC, AB
[2.2], Check1, D2A BOH, SD
[63], name2, MFB, k
This
split($3, arr, / +/)splits column#3 into the arrayarrbased on space separator (there may one-or-more spaces will be there as separator).In
for(val in arr) !uniq_arr[ arr[val] ]++, we are creating a new array with removed duplicated values taken from arrayarr; so the final values in arrayuniq_arrare only unique values in each column3.next we are just printing saved values in
uniq_arrand delete that key after it printed; note that values of column#1, #2 & #4 were printed separately.
see also:
- How does awk '!a[$0]++' work?
- remove duplicated pattern/entries within each field in CSV file
answered yesterday
αғsнιηαғsнιη
17.2k103069
add a comment |
Uniq -f option helps you please check
answered yesterday
AnavarathanAnavarathan
205
1
Mine question is bit different, that to be check only column3 and make it uniq . Other columns will remain intact
– Jack15
yesterday
Waiting for other answers....
– Jack15
yesterday
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "106"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Jack15 is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f511825%2fremove-duplicates-from-particular-columns%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
with awk:
awk -F, '
printf $1 FS $2 FS;
split($3, arr, / +/); for(val in arr) !uniq_arr[ arr[val] ]++;
for (key in uniq_arr)
printf (key!="")? SPACE key:""; SPACE=" "; delete uniq_arr[key]
;
printf FS $4"n"
' infile
[1], Value1, UAC, AB
[2.2], Check1, D2A BOH, SD
[63], name2, MFB, k
This
split($3, arr, / +/)splits column#3 into the arrayarrbased on space separator (there may one-or-more spaces will be there as separator).In
for(val in arr) !uniq_arr[ arr[val] ]++, we are creating a new array with removed duplicated values taken from arrayarr; so the final values in arrayuniq_arrare only unique values in each column3.next we are just printing saved values in
uniq_arrand delete that key after it printed; note that values of column#1, #2 & #4 were printed separately.
see also:
- How does awk '!a[$0]++' work?
- remove duplicated pattern/entries within each field in CSV file
answered yesterday
αғsнιηαғsнιη
17.2k103069
add a comment |
with awk:
awk -F, '
printf $1 FS $2 FS;
split($3, arr, / +/); for(val in arr) !uniq_arr[ arr[val] ]++;
for (key in uniq_arr)
printf (key!="")? SPACE key:""; SPACE=" "; delete uniq_arr[key]
;
printf FS $4"n"
' infile
[1], Value1, UAC, AB
[2.2], Check1, D2A BOH, SD
[63], name2, MFB, k
This
split($3, arr, / +/)splits column#3 into the arrayarrbased on space separator (there may one-or-more spaces will be there as separator).In
for(val in arr) !uniq_arr[ arr[val] ]++, we are creating a new array with removed duplicated values taken from arrayarr; so the final values in arrayuniq_arrare only unique values in each column3.next we are just printing saved values in
uniq_arrand delete that key after it printed; note that values of column#1, #2 & #4 were printed separately.
see also:
- How does awk '!a[$0]++' work?
- remove duplicated pattern/entries within each field in CSV file
answered yesterday
αғsнιηαғsнιη
17.2k103069
add a comment |
with awk:
awk -F, '
printf $1 FS $2 FS;
split($3, arr, / +/); for(val in arr) !uniq_arr[ arr[val] ]++;
for (key in uniq_arr)
printf (key!="")? SPACE key:""; SPACE=" "; delete uniq_arr[key]
;
printf FS $4"n"
' infile
[1], Value1, UAC, AB
[2.2], Check1, D2A BOH, SD
[63], name2, MFB, k
This
split($3, arr, / +/)splits column#3 into the arrayarrbased on space separator (there may one-or-more spaces will be there as separator).In
for(val in arr) !uniq_arr[ arr[val] ]++, we are creating a new array with removed duplicated values taken from arrayarr; so the final values in arrayuniq_arrare only unique values in each column3.next we are just printing saved values in
uniq_arrand delete that key after it printed; note that values of column#1, #2 & #4 were printed separately.
see also:
- How does awk '!a[$0]++' work?
- remove duplicated pattern/entries within each field in CSV file
answered yesterday
αғsнιηαғsнιη
17.2k103069
with awk:
awk -F, '
printf $1 FS $2 FS;
split($3, arr, / +/); for(val in arr) !uniq_arr[ arr[val] ]++;
for (key in uniq_arr)
printf (key!="")? SPACE key:""; SPACE=" "; delete uniq_arr[key]
;
printf FS $4"n"
' infile
[1], Value1, UAC, AB
[2.2], Check1, D2A BOH, SD
[63], name2, MFB, k
This
split($3, arr, / +/)splits column#3 into the arrayarrbased on space separator (there may one-or-more spaces will be there as separator).In
for(val in arr) !uniq_arr[ arr[val] ]++, we are creating a new array with removed duplicated values taken from arrayarr; so the final values in arrayuniq_arrare only unique values in each column3.next we are just printing saved values in
uniq_arrand delete that key after it printed; note that values of column#1, #2 & #4 were printed separately.
see also:
- How does awk '!a[$0]++' work?
- remove duplicated pattern/entries within each field in CSV file
answered yesterday
αғsнιηαғsнιη
17.2k103069
answered yesterday
αғsнιηαғsнιη
17.2k103069
answered yesterday
αғsнιηαғsнιη
17.2k103069
answered yesterday
αғsнιηαғsнιη
17.2k103069
17.2k103069
add a comment |
add a comment |
Uniq -f option helps you please check
answered yesterday
AnavarathanAnavarathan
205
1
Mine question is bit different, that to be check only column3 and make it uniq . Other columns will remain intact
– Jack15
yesterday
Waiting for other answers....
– Jack15
yesterday
add a comment |
Uniq -f option helps you please check
answered yesterday
AnavarathanAnavarathan
205
1
Mine question is bit different, that to be check only column3 and make it uniq . Other columns will remain intact
– Jack15
yesterday
Waiting for other answers....
– Jack15
yesterday
add a comment |
Uniq -f option helps you please check
answered yesterday
AnavarathanAnavarathan
205
Uniq -f option helps you please check
answered yesterday
AnavarathanAnavarathan
205
answered yesterday
AnavarathanAnavarathan
205
answered yesterday
AnavarathanAnavarathan
205
answered yesterday
AnavarathanAnavarathan
205
205
1
Mine question is bit different, that to be check only column3 and make it uniq . Other columns will remain intact
– Jack15
yesterday
Waiting for other answers....
– Jack15
yesterday
add a comment |
1
Mine question is bit different, that to be check only column3 and make it uniq . Other columns will remain intact
– Jack15
yesterday
Waiting for other answers....
– Jack15
yesterday
1
1
Mine question is bit different, that to be check only column3 and make it uniq . Other columns will remain intact
– Jack15
yesterday
Mine question is bit different, that to be check only column3 and make it uniq . Other columns will remain intact
– Jack15
yesterday
Waiting for other answers....
– Jack15
yesterday
Waiting for other answers....
– Jack15
yesterday
add a comment |
Jack15 is a new contributor. Be nice, and check out our Code of Conduct.
Jack15 is a new contributor. Be nice, and check out our Code of Conduct.
Jack15 is a new contributor. Be nice, and check out our Code of Conduct.
Jack15 is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f511825%2fremove-duplicates-from-particular-columns%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
-awk, csv, sed, text-processing
order of entries after removing supplicates in 3rd column is matter?
– αғsнιη
yesterday
no matter, But contents should be in same line.
– Jack15
yesterday