Gunjan's blog
Day 10 : Log Analyzer and Report Generator

Day 10 : Log Analyzer and Report Generator

Gunjan Bhadade's photo
Gunjan Bhadade
·

4 min read

Table of contents

Hey DevOps Enthusiasts! 👋

Welcome to Day 10 of our 90DaysofDevOps journey. Today, we are diving into a crucial task for any system administrator: log analysis and report generation. Logs are essential for monitoring and troubleshooting servers, and automating the analysis process can save a lot of time and effort. Let's get started!

Challenge Description 📝

As a system administrator, you need to analyze log files daily, identify specific events, and generate summary reports. We'll create a bash script that performs the following tasks:

  1. Input: Takes the path to the log file as a command-line argument.

  2. Error Count: Counts the number of error messages in the log file.

  3. Critical Events: Searches for "CRITICAL" events and prints those lines with line numbers.

  4. Top Error Messages: Identifies and displays the top 5 most common error messages.

  5. Summary Report: Generates a summary report with details like the date of analysis, log file name, total lines processed, total error count, top 5 error messages, and critical events.

The Script 🛠️

Here's the bash script to automate the log analysis and report generation:

#!/bin/bash

log_file=$1

# Check if the log file exists
if [ ! -f "$log_file" ]; then
    echo "Log file not found!"
    exit 1
fi

# Variables
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
report_file="log_summary_$(date '+%Y%m%d').txt"
error_keyword="ERROR"
critical_keyword="CRITICAL"

# Error Count
error_count=$(grep -c "$error_keyword" "$log_file")

# Critical Events
critical_events=$(grep -n "$critical_keyword" "$log_file")

# Top 5 Error Messages
declare -A error_messages
while read -r line; do
    if [[ "$line" == *"$error_keyword"* ]]; then
        error_message=$(echo "$line" | awk -F"$error_keyword" '{print $2}')
        ((error_messages["$error_message"]++))
    fi
done < "$log_file"

# Sort and get top 5 error messages
top_errors=$(for key in "${!error_messages[@]}"; do echo "${error_messages[$key]} $key"; done | sort -rn | head -5)

# Summary Report
total_lines=$(wc -l < "$log_file")
{
    echo "Log Analysis Report - $timestamp"
    echo "Log File: $log_file"
    echo "Total Lines Processed: $total_lines"
    echo "Total Error Count: $error_count"
    echo
    echo "Top 5 Error Messages:"
    echo "$top_errors"
    echo
    echo "Critical Events:"
    echo "$critical_events"
} > "$report_file"

# Print the report location
echo "Summary report generated: $report_file"

# Optional: Move the log file to an archive directory
archive_dir="./archive"
mkdir -p "$archive_dir"
mv "$log_file" "$archive_dir"

echo "Log file archived to: $archive_dir"

OUTPUT :

Explanation 📖

Initial Setup

log_file=$1
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
report_file="log_summary_$(date '+%Y%m%d').txt"
error_keyword="ERROR"
critical_keyword="CRITICAL"

  • log_file: The path to the log file, provided as a command-line argument.

  • timestamp: The current date and time for the report.

  • report_file: The name of the summary report file.

  • error_keyword and critical_keyword: Keywords to search for errors and critical events.

Error Count

error_count=$(grep -c "$error_keyword" "$log_file")
  • grep -c "$error_keyword" "$log_file": Counts the number of lines containing the keyword "ERROR".

Critical Events

critical_events=$(grep -n "$critical_keyword" "$log_file")
  • grep -n "$critical_keyword" "$log_file": Searches for lines containing "CRITICAL" and prints those lines with line numbers.

Top 5 Error Messages

declare -A error_messages
while read -r line; do
    if [[ "$line" == *"$error_keyword"* ]]; then
        error_message=$(echo "$line" | awk -F"$error_keyword" '{print $2}')
        ((error_messages["$error_message"]++))
    fi
done < "$log_file"

  • declare -A error_messages: Declares an associative array to store error messages and their counts.

  • while read -r line: Reads each line of the log file.

  • if [[ "$line" == "$error_keyword" ]]: Checks if the line contains "ERROR".

  • error_message=$(echo "$line" | awk -F"$error_keyword" '{print $2}'): Extracts the error message.

  • ((error_messages["$error_message"]++)): Increments the count of the error message in the array.

Sort and Get Top 5 Error Messages

top_errors=$(for key in "${!error_messages[@]}"; do echo "${error_messages[$key]} $key"; done | sort -rn | head -5)

  • for key in "${!error_messages[@]}": Iterates over the keys (error messages) in the array.

  • echo "${error_messages[$key]} $key": Prints the count and the error message.

  • sort -rn: Sorts the output in reverse numerical order.

  • head -5: Gets the top 5 error messages.

Summary Report

total_lines=$(wc -l < "$log_file")
{
    echo "Log Analysis Report - $timestamp"
    echo "Log File: $log_file"
    echo "Total Lines Processed: $total_lines"
    echo "Total Error Count: $error_count"
    echo
    echo "Top 5 Error Messages:"
    echo "$top_errors"
    echo
    echo "Critical Events:"
    echo "$critical_events"
} > "$report_file"

  • total_lines=$(wc -l < "$log_file"): Counts the total number of lines in the log file.

  • echo "...": Prints the report details.

  • \> "$report_file": Redirects the output to the report file.

Archive the Log File

archive_dir="./archive"
mkdir -p "$archive_dir"
mv "$log_file" "$archive_dir"

  • mkdir -p "$archive_dir": Creates the archive directory if it doesn't exist.

  • mv "$log_file" "$archive_dir": Moves the log file to the archive directory.

Conclusion 🎉

Today, we learned how to create a bash script to automate the analysis of log files and generate summary reports. This is a vital skill for any system administrator, helping to monitor system health and troubleshoot issues efficiently. Keep practicing and refining your scripting skills. See you on Day 11 of our DevOps journey! 🚀✨

Happy scripting! 😊

bash script#90daysofdevops