Ttyjfyk

Below is how a log looks like:

Company=XYZ
Req_id=1234
Time_taken=10 sec
Status=Success

Company=ABC
Req_id=3456
Time_taken=200 sec
Status=Failure

Company=DFG
Req_id=3001
Time_taken=15 sec
Status=Success

We have to get top 3 request IDs where max time is taken.

I tried with below solution where I get request id and time taken but I am not happy with my answer:

awk -vRS= -F'[=n]' '/Time_taken/print $4,$6' test.txt | sort -nr

How could I have done this better? Use some logic to code instead of inbuilt functions. Also, could someone help me to understand -F'[=n]' better? I have been just copying it from my previous scripts.

Output should be:

Below Request Id took more then expected
Request id 3456, Time Taken 200 sec
Request id 3001, Time Taken 15 sec 
Request id 1234, Time Taken 10 sec

We can do this using the paragraph mode of Perl wherein we read in paragraph sized chunk of files as records and build up a hash keyed on IDs and values as time taken. At the end of file, we reverse sort numerically on the times taken and then print the desired message.

$ perl -ln -00 -e '
 %h = (%h, /^Req_id=(d+)n.*^Time_taken=(d+)/ms){
 print "The below IDs took more than expected:" if scalar keys %h;
 print join " ", "Req ID:", "$_," , "Time taken", $h$_, "sec"
 for (sort $h$b <=> $h$a keys %h)[0..2];
' input.file

Output:

The below IDs took more than expected:
Req ID: 3456, Time taken 200 sec
Req ID: 3001, Time taken 15 sec
Req ID: 1234, Time taken 10 sec

you need to remember request number.

I would use (this could be one lined)

awk -F= '$1 == "Req_id" r=$2 ; 
 $1 == "Time_taken" printf "Request id %s %s %sn",r,$1,$2 ; ' file |
sort -r -n -k5 |
head -3

which give

Request id 3456 Time_taken 200 sec
Request id 3001 Time_taken 15 sec
Request id 1234 Time_taken 10 sec

where

• 
  -F= use = as separator


• 
  $1 == "Req_id" r=$2 ; get last request id


• 
  $1 == "Time_taken" if line is "time taken"


• 
  printf "Request id %s %s %sn",r,$1,$2 ; print request id and seconds


• 
  | sort pipe to sort


• 
  -r reverse order


• 
  -n numeric sort (e.g. 200 greater than 15)


• 
  -k5 on 5th field


• 
  | head -3 get first 3 lines

I would prefer Python over awk for anything except one-liners. Using a Python script will allow you to more easily handle poorly formatted input and to perform logging and error-handling.

Here is a basic Python script that will produce the desired output for your example input, and which has a structure which is suggestive of how you might build it out further if needed:

#!/usr/bin/env python3
# -*- encoding: utf-8 -*-
"""parse_log.py"""

import sys
from collections import OrderedDict

logfilepath = sys.argv[1]

# Define a function to parse a single block/entry in the log file
def parse_block(block):
 parsed_block = dict()
 lines = block.split("n")
 for line in lines:
 if line.startswith("Company="):
 parsed_block["Company"] = line[8:]
 elif line.startswith("Req_id="):
 parsed_block["Required_ID"] = line[7:]
 elif line.startswith("Time_taken="):
 parsed_block["Time_Taken"] = line[11:]
 elif line.startswith("Status="):
 parsed_block["Status"] = line[7:]
 else:
 pass
 return parsed_block

# Initialize a list to store the processed entries
parsed_blocks = list()

# Populate the list
with open(logfilepath, "r") as logfile:
 blocks = logfile.read().split("nn")
 for block in blocks:
 parsed_block = parse_block(block)
 parsed_blocks.append(parsed_block)

# Print the results
print("Below Request Id took more then expected")
for parsed_block in parsed_blocks:
 print("Request id , Time Taken: ".format(parsed_block["Required_ID"], parsed_block["Time_Taken"]))

You could run it like this:

python parse_log.py data.log

On your example input, it produces the following output (as requested):

Below Request Id took more then expected
Request id 1234, Time Taken: 10 sec
Request id 3456, Time Taken: 200 sec
Request id 3001, Time Taken: 15 sec