Ttyjfyk

My end goal is to have a script that will count the instances of each username in all files.

A username is a string, in quotes, that follows the string 'login'. For example, in one file, I might have:

"this":"is', "a":"strange", "type":"of":"object", "but":"please",
 "go":"withit", "login":"username1"

"this":"is', "login":"username2", "type":"of":"object", "but":"please",
 "go":"withit"

And in another file, I might have:

"this":"is', "a":"strange", "type":"of":"object", "but":"please",
 "go":"withit", "login":"username3"

"login":"username1", "please":"gowithit"

In which case, I'd like to have a txt file that contains a dict object with the count of the number of times each username appears in the files:

"username1": 2, "username2":1, "username3":1

I've read a few things to get me started, but I can't seem to put this together. I've sort of pseudocoded it, but I can't progress from this point.

I think I need to do this in two stages.

1) Get a list of all the usernames

2) Count the number of times each username appears in all files.

For task 1):

 grep 'login:' * | sed 's/^.*: //'
#Except I think this gets everything from the line after 'login', which isn't what I want.

For task 2):

for all_usernames_in_file:
 stringval = username_read_from_saved_file
 cat * | grep -c $stringval > output.txt

Can anyone take it from here?

EDIT:

Do you mean I should do this:

grep -o 'login":"[^"]*"' /path/to/dir/* | cut -d'"' -f3 | sort | uniq -c | sed '1i s/s*([0-9]*)s*(.*)/"2": 1,/;$a' > output.txt

EDIT 2: Still not working. I'm trying to diagnose by understanding what each command does.

Let's say I'm just looking at this part to start:

grep -o 'login":"[^"]*"' /path/to/dir/* | cut -d'"' -f3 | sort | uniq -c > myfile.txt

Right now, myfile.txt is blank.

Here's what I think this command is doing:

grep -o matches non-empty parts of a matching line.

'login":"[^"]*"' is the string we want grep to match. In the middle, the [^"] matches any character after login":" not equal to ", and the * says we want any length of match - that is, the length of the username doesn't matter, we want everything between the quotes.

| is a pipe. It means "and then"

cut -d '"' -f3 means slice up the returned line (all stuff after login":"), using the delimiter ", and take field 3 (that is, just the username).

| is a pipe. It means "and then"

sort the usernames

| is a pipe. It means "and then"

Get the unique usernames and count the number of times each appears.

If I take that much, and put a > myfile.txt at the end, then I should end up with a txt file that contains usernames and a count of the number of times each appears. It won't be well-formatted, but it will exist.

Why am I not getting such a file?

NOTE: does it matter that I'm searching through .json.gz formatted files? I've gotten the script to work when searching through txt, but not through the other format.

@rush use of sed didn't work in my shell so I went this way

grep -Poh '(?<=login":")[^"]*' json* | sort | uniq -c | awk -v OFS=': ' 'BEGINprint ""print $2, $1ENDprint""' | sed -E 's/([0-9])$/1,/g;s/:/":/g;s/^([^])/"1/g'

The multiple sed can be amended if your shell lets you escape the " and print them in the awk statement.

grep -Poh '(?<=login":")[^"]*' json* | sort | uniq -c | awk -v OFS=': ' 'BEGINprint ""print "$2", $1ENDprint""' | sed -E 's/([0-9])$/1,/g'

In my shell awk choked on the " in the second script. Not sure why but am sure that someone out there will tell me.

I also tried jq but it choked on the json files. There appears to be a syntax error

"this":"is' #is written so I edited these to
"this":"is"

Also jq didn't like the construct

"a":"strange" # so I also edited these to
b: "a":"strange"

If original files are supposed to be as per the edits made then jq works

jq '.login' json* | sort | uniq -c | awk -v OFS=': ' 'BEGINprint ""print $2, $1ENDprint""' | sed -E 's/([0-9])$/1,/g'