Ttyjfyk

I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so that I can use the pure matrix information in another program. How can I use bash shell commands to do this?

If it's any help, I'm using RedHat and an Ubuntu linux systems.

As long as the file is not a symlink or hardlink, you can use sed, tail, or awk. Example below.

$ cat t.txt
12
34
56
78
90

sed

$ sed -e '1,3d' < t.txt
78
90

You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile. This won't echo anything, it will just modify the file in-place. If you don't need to pipe the result to another command, this is easier.

tail

$ tail -n +4 t.txt
78
90

awk

$ awk 'NR > 3 print ' < t.txt
78
90

sed -i '1,3d' file.txt

This deletes first 3 line from file.txt.

If the tabulated lines are the ones that have a tab character:

grep '␉' <input_file >output_file

(␉ being a literal tab character) or equivalently

sed -n '/␉/p' <input_file >output_file

In a bash/ksh/zsh script, you can write $'t' for a tab, e.g. grep $'t' or sed -n $'/t/p'.

If you want to eliminate 10 lines at the beginning of the file:

tail -n +11 <input_file >output_file

(note that it's +11 to eliminate 10 lines, because +11 means “start from line 11” and tail numbers lines from 1) or

sed '1,10d' <input_file >output_file

On Linux, you can take advantage of GNU sed's -i option to modify files in place:

sed -i -n '/t/p' *.txt

Or you can use a shell loop and temporary files:

for x in *.txt; do
 tail -n +11 <"$x" >"$x.tmp"
 mv "$x.tmp" "$x"
done

Or if you don't want to modify the files in place, but instead give them a different name:

for x in *.txt; do
 tail -n +11 <"$x" >"$x%.txt.data"
done

You can use Vim in Ex mode:

ex -sc '1d5|x' file

1. 1 move to first line




2. 5 select 5 lines




3. d delete




4. x save and close

echo "anb" | sed '1d' # deletes first line

cat list.txt | sed '1d' > list.csv # read list.txt and write list.csv without first line

Other useful commands:

grep '^|' # finds first character (pipe|)

sed 's/|//g' # deletes pipe

sed 's/ //g' # deletes space

By percentage

Using bash, to clean up a file using a percentage number instead of an absolute number of lines:

sed -i -e 1,$( printf "$((`cat php_errors.log | wc -l` * 75 /100 ))" )d php_errors.log

Watch out because that command can be destructive since it deletes content in-place, without creating a copy.

It deletes the first 75% of lines from the mentioned file.

I am a bash newbie, so I am sure this could be made prettier and faster, and I welcome suggestions to achieve this.