Building an AI Log Inspector with Rust OpenAI and Homebrew

Motivation

After writing extensively about data engineering, I wanted to explore practical AI applications. The goal is to build a simple yet useful tool that leverages AI for log analysis and is packaged as a local command-line utility. This combines my interest in AI exploration with the practicality of daily engineering tasks.

Solution Overview

• AI-powered log analysis using OpenAI's GPT models

• Smart chunking for handling large log files

• Automated error classification and summarization

• Distributed via Homebrew for easy installation

Part 1: OpenAI Integration

Log analysis is a critical but time-consuming task. Engineers often spend hours scanning through logs to identify patterns, errors, and root causes. ChatGPT excels at natural language processing and pattern recognition, making it perfect for this use case.

Key benefits of using ChatGPT for log analysis:

• Instant error classification and categorization

• Natural language summaries of complex log patterns

• Context-aware analysis across log segments

• Ability to spot subtle relationships in log entries

For this tool, we're using GPT-3.5-turbo with carefully crafted prompts to:

• Classify errors and their severity

• Generate concise summaries of log segments

• Handle large log files efficiently through smart chunking

Let's dive into the implementation:

Smart Chunking Strategy

When working with log files, we face two key challenges:

• Log files can be very large (GBs)

• OpenAI API has token limits (4096 tokens for GPT-3.5-turbo)

Our solution: highlights from chunk_strategy.rs:

pub struct ChunkStrategy {
    max_tokens_per_chunk: usize,    // 2500 tokens - safe margin for GPT-3.5
    overlap_lines: usize,           // 5 lines overlap for context
    context_window: usize,          // 1000 chars for boundary analysis
    estimated_line_size: usize,     // 100 bytes average
    max_lines_per_chunk: usize,     // Cap at 200 lines
}

Key features:

• Automatic chunk size calculation based on file size

• Smart boundary detection at log entry breaks

• Overlap handling to maintain context between chunks

• Token limit compliance for OpenAI API

OpenAI Client

The OpenAI client needs to handle two core tasks:

1. Structured communication with OpenAI API

2. Converting log analysis into actionable insights

Here's how it works: key implementation from openai_client.rs:

pub struct OpenAIClient {
    api_key: String,
    host: String,
    client: reqwest::Client,
}

// Generic chat method for different analysis types
pub async fn chat<T>(&self, prompt: &str, content: &str) -> Result<T, Box<dyn StdError>>
where T: DeserializeOwned {
    // Structured message format
    let messages = vec![
        Message { role: "system", content: prompt.to_string() },
        Message { role: "user", content: content.to_string() }
    ];
}

Features:

• Generic response handling for different analysis types

• Consistent prompt structure

• Error handling and retries

• JSON response parsing

Log Inspector Core

This is where the magic happens! The Log Inspector combines our chunking strategy and OpenAI client to deliver intelligent log analysis.

Key implementation from log_inspector.rs:

pub struct LogInspector {
    client: OpenAIClient,
}

impl LogInspector {
    pub async fn error_classify(&self, content: &str) -> Result<String> {
        // Smart error pattern detection
    }

    pub async fn summarize(&self, content: &str) -> Result<String> {
        // Context-aware summarization
    }
}

Features:

• Parallel chunk processing

• Intelligent error categorization

• Context-preserving summaries

• Structured output format

Part 2: Distribution with Homebrew

Time to make our AI-powered tool easily accessible! Homebrew provides the perfect distribution channel for macOS users.

Release automation flow:

graph TD
    A[Version Update] --> B[GitHub Action]
    B --> C[Build Binary]
    C --> D[Calculate SHA256]
    D --> E[Update Formula]
    E --> F[Create PR]
    F --> G[Auto-merge]

Key workflow from release.yml:

name: Release
on:
  push:
    paths: ['VERSION']
    branches: [main]

jobs:
  release:
    runs-on: macos-latest
    steps:
      - name: Build
        run: cargo build --release
      - name: Update Formula
        run: |
          SHA256=$(shasum -a 256 target/release/log-inspector | cut -d ' ' -f 1)
          sed -i '' "s/sha256 \".*\"/sha256 \"$SHA256\"/" Formula/log-inspector.rb

Installation and Usage

Getting started with Log Inspector is straightforward:

# Install via Homebrew
brew tap hoaihuongbk/log-inspector
brew install log-inspector

Configuration setup:

# Create config file with your OpenAI key
echo "openai_api_key = your_key_here" > ~/.log-inspector.conf

Run analysis:

# Analyze any log file
log-inspector example.log
Starting log inspection for: example.log

=== Processing chunk 1 ===
Error Codes: NETWORK_ERROR, SPARK_ERROR, UNKNOWN_ERROR
Summary: Overall situation: The application started successfully but encountered errors related to database connection, high memory usage, and connection timeouts shortly after.

- Database connection error occurred at 10:15:31
- High memory usage warning was logged at 10:15:32
- Connection timeout error was encountered at 10:15:33

Key features in action:

• Fast error classification with GPT-3.5-turbo

• Smart chunking for efficient processing

• Context-aware summarization

• Clean, readable output

Key Learnings and Future Improvements

What we've achieved:

• Built a practical AI-powered log analysis tool

• Implemented efficient chunking for large log files

• Leveraged GPT-3.5-turbo for smart log interpretation

• Created an automated release pipeline with Homebrew

Technical highlights:

• Rust for performance and reliability

• OpenAI API for intelligent analysis

• GitHub Actions for seamless deployment

• Homebrew for easy distribution

Future improvements:

• Enhanced chunking algorithm

• Parallel processing implementation

• Colorful terminal output, rich formatting with icons

Full code available at: github.com/hoaihuongbk/log-inspector

Let's build more tools that combine AI capabilities with practical engineering needs! 🚀