Analyzing Kubernetes Audit Logs

Parses Kubernetes API server audit logs (JSON lines) to detect exec-into-pod, secret access, RBAC modifications,

Category: development Source: mukul975/Anthropic-Cybersecurity-Skills

Analyzing Kubernetes Audit Logs

Kubernetes audit logs are a critical source of visibility into the actions performed within a Kubernetes cluster. These logs, typically emitted by the Kubernetes API server, record every API request made to the cluster, providing a detailed, chronological account of user and system activity. The "Analyzing Kubernetes Audit Logs" skill is designed for cybersecurity professionals, SOC analysts, and platform engineers who need to identify suspicious or unauthorized activities in Kubernetes environments. This article provides an in-depth guide to understanding, using, and applying this skill effectively.

What Is This Skill?

The "Analyzing Kubernetes Audit Logs" skill parses Kubernetes API server audit logs (in JSON lines format) to detect a variety of security-relevant events. Specifically, it automates the identification of:

Exec-into-pod operations (such as kubectl exec)
Access to Kubernetes secrets and configmaps
Modifications to RBAC (Role-Based Access Control) policies
Creation of privileged pods
Anonymous or unauthenticated API access

The skill is implemented in Python and is intended to help analysts build detection rules and threat hunting queries based on observed audit event patterns. It is particularly useful for incident investigations, validating SIEM/SOC monitoring coverage, and strengthening response playbooks for Kubernetes-specific threats.

Why Use This Skill?

Kubernetes clusters are high-value targets for attackers due to their role in orchestrating containerized workloads and managing sensitive credentials. However, Kubernetes' distributed and dynamic nature makes traditional security monitoring approaches insufficient. The audit logs contain granular details about every API interaction, including who did what, when, and from where.

This skill automates the time-consuming process of manually sifting through audit logs by:

Extracting and highlighting security-relevant events
Enabling rapid detection of lateral movement, privilege escalation, and data exfiltration techniques
Supporting the creation of custom detection rules for SIEM platforms
Providing structured procedures for SOC analysts to follow during investigations

By leveraging this skill, organizations can improve their ability to detect, investigate, and respond to Kubernetes-specific threats and misconfigurations.

How to Use This Skill

Prerequisites

Before utilizing this skill, ensure you have:

A basic understanding of Kubernetes concepts, especially around API operations and RBAC
Access to Kubernetes audit log files (typically in /var/log/kubernetes/audit.log or a centralized logging system)
Python 3.8 or higher with necessary dependencies installed
Appropriate authorization to analyze audit logs, especially in production or sensitive environments

Parsing Audit Logs in Python

The audit logs are formatted as JSON objects, each representing a single API server event. Below is a basic example of iterating through an audit log file in Python:

import json

with open("/var/log/kubernetes/audit.log") as f:
    for line in f:
        event = json.loads(line)
        # Analyze the event here

Detecting Security-Relevant Events

The skill applies specific detection logic for common attack vectors:

Exec-Into-Pod Detection
Identify when a user executes a command in a running pod, which could indicate lateral movement or post-exploitation.

if event.get("objectRef", {}).get("resource") == "pods" and \
   event.get("verb") == "create" and \
   event.get("requestURI", "").endswith("/exec"):
    print(f"Exec into pod detected: {event['user']['username']} in {event['objectRef']['namespace']}")

Secret Access Monitoring
Detect API calls that read or list secrets, which may indicate attempts to exfiltrate sensitive data.

if event.get("objectRef", {}).get("resource") == "secrets" and \
   event.get("verb") in ["get", "list"]:
    print(f"Secret accessed: {event['user']['username']} in {event['objectRef']['namespace']}")

RBAC Modification Detection
Catch changes to Roles, RoleBindings, ClusterRoles, and ClusterRoleBindings, which could be used for privilege escalation.

rbac_resources = ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
if event.get("objectRef", {}).get("resource") in rbac_resources and \
   event.get("verb") in ["create", "update", "patch", "delete"]:
    print(f"RBAC modified: {event['user']['username']} - {event['verb']}")

Privileged Pod Creation
Detect when pods are created with elevated privileges, which is often an indicator of an attack.

(This detection requires inspecting the pod spec in the requestObject field.)

if event.get("objectRef", {}).get("resource") == "pods" and \
   event.get("verb") == "create":
    pod_spec = event.get("requestObject", {}).get("spec", {})
    containers = pod_spec.get("containers", [])
    for container in containers:
        security_ctx = container.get("securityContext", {})
        if security_ctx.get("privileged") is True:
            print("Privileged pod creation detected.")

Anonymous API Access
Identify events where the API request was made by an unauthenticated (anonymous) user.

if event.get("user", {}).get("username") == "system:anonymous":
    print("Anonymous API access detected.")

Building Detection Rules

The patterns above can be adapted into SIEM queries or automated scripts for continuous monitoring. This skill encourages the creation of reusable detection logic tailored to your cluster’s needs.

When to Use It

Incident Response: When analyzing potential or confirmed Kubernetes cluster compromises
Threat Hunting: When searching for TTPs (Tactics, Techniques, and Procedures) associated with known attack patterns
Detection Engineering: When developing SIEM rules for Kubernetes-specific attack surfaces
Security Validation: When verifying that monitoring and logging controls are effective against realistic attack scenarios

Important Notes

Log Volume: Kubernetes audit logs can be extremely verbose. Filtering for security-relevant events using the approaches above is essential for efficiency.
Audit Policy Configuration: The effectiveness of detection is contingent on your Kubernetes audit policy. Ensure the policy is configured to log relevant events at an appropriate level (at least Metadata or Request).
Data Sensitivity: Audit logs may contain sensitive information, including usernames, commands, and secrets. Handle these files with care and restrict access as necessary.
Performance Considerations: Processing large log files may require batching or streaming approaches for scalability.
Continuous Improvement: As Kubernetes evolves, periodically review and update detection rules to cover new API resources or attack techniques.

By leveraging the "Analyzing Kubernetes Audit Logs" skill, security teams can enhance their visibility, detection, and response capabilities within Kubernetes environments, ensuring proactive defense against evolving container threats. For more details and source code, refer to the GitHub repository.