Configuring Host-Based Intrusion Detection

What Is This

Configuring Host-Based Intrusion Detection is a cybersecurity skill that involves deploying and managing software systems specifically designed to monitor individual computers (endpoints) for signs of compromise, unauthorized changes, or policy violations. Host-Based Intrusion Detection Systems (HIDS) such as OSSEC, Wazuh, and AIDE operate on endpoints to analyze file integrity, system call activity, configuration changes, and user actions. Unlike network-based intrusion detection systems, HIDS focuses on monitoring the internal state of each host, providing detailed visibility into suspicious or malicious activities that may bypass perimeter defenses.

This skill is vital for IT and security professionals responsible for maintaining the integrity and security of servers, workstations, and cloud instances. Typical use cases include deploying agents across Windows and Linux environments, configuring File Integrity Monitoring (FIM) for compliance requirements, integrating HIDS alerts with Security Information and Event Management (SIEM) platforms, and responding to detected anomalies or security policy violations.

Why Use It

The primary reasons to configure host-based intrusion detection include:

1. Early Threat Detection: HIDS can identify attacks such as rootkits, malware, unauthorized file changes, and suspicious user behavior at the host level before they escalate into major incidents.
2. Compliance Requirements: Regulations like PCI DSS (Requirement 11.5), HIPAA, and NIST CSF mandate file integrity monitoring and change detection on critical systems.
3. Change Management: Monitoring and alerting on unauthorized or unexpected changes to system files, configurations, and binaries helps maintain environment integrity.
4. Centralized Security Monitoring: When integrated with a SIEM, HIDS provides real-time visibility and alerting across distributed endpoints, improving incident response capabilities.
5. Forensic Investigations: HIDS logs and alerts serve as valuable evidence during security investigations, enabling root cause analysis and compliance reporting.

How to Use It

1. Deploying a HIDS

Agent (Wazuh Example)

Prerequisites:

• Wazuh manager (server) is installed and reachable from endpoints.
• Administrative privileges on the endpoint.
• Network ports 1514 (TCP/UDP) open between agent and manager.

Installation on Linux (Ubuntu/Debian):

curl -sO https://packages.wazuh.com/4.x/apt/wazuh-agent_4.4.5-1_amd64.deb
sudo dpkg -i ./wazuh-agent_4.4.5-1_amd64.deb
sudo systemctl enable wazuh-agent
sudo systemctl start wazuh-agent

Configuration (/var/ossec/etc/ossec.conf):

<ossec_config>
  <client>
    <server-ip>10.0.0.10</server-ip>   <!-- Wazuh manager IP -->
  </client>
  <syscheck>
    <frequency>3600</frequency>
    <directories check_all="yes">/etc,/bin,/usr/bin,/sbin</directories>
  </syscheck>
</ossec_config>

Register Agent with Manager:

/var/ossec/bin/agent-auth -m 10.0.0.10

2. Configure File Integrity

Monitoring (FIM)

FIM detects unauthorized or unexpected changes to critical files and directories.

Example (OSSEC/Wazuh syscheck block):

<syscheck>
  <frequency>3600</frequency>
  <directories check_all="yes">/etc,/usr/bin,/sbin</directories>
  <ignore>/etc/mtab</ignore>
  <alert_new_files>yes</alert_new_files>
</syscheck>

3. Monitoring Configuration Changes and Security

Violations

Customize rules and policies to monitor system configurations, registry changes (on Windows), and user activity. For example, enable rootkit detection and policy violation alerts within ossec.conf.

4. Integrating with

SIEM

Forward HIDS alerts to a SIEM for centralized monitoring and correlation. Configure Wazuh or OSSEC to send alerts to syslog or directly to platforms like Splunk or Elastic Stack.

Example (output block):

<ossec_config>
  <remote>
    <connection>syslog</connection>
    <port>514</port>
    <protocol>udp</protocol>
    <allowed-ips>10.0.0.100</allowed-ips> <!-- SIEM IP -->
  </remote>
</ossec_config>

When to Use It

Leverage the Configuring Host-Based Intrusion Detection skill when:

• Deploying HIDS agents (such as Wazuh, OSSEC, or AIDE) to Windows and Linux endpoints to monitor for local suspicious activity.
• Setting up file integrity monitoring to meet compliance standards such as PCI DSS 11.5 or NIST SI-7.
• Detecting unauthorized system or configuration changes, rootkit attempts, and policy violations on hosts.
• Integrating HIDS with SIEM solutions for real-time, centralized monitoring and alerting.
• Building custom FIM policies tailored to your organization's critical files and directories.

Avoid using this skill for network-based intrusion detection (such as Suricata or Snort), or for deploying endpoint detection and response (EDR) platforms, as the focus here is strictly host-based.

Important Notes

• Performance Impact: HIDS agents can consume system resources, especially during large-scale file integrity checks. Tune directories and frequency settings to balance performance and coverage.
• False Positives: Initial configuration may generate many alerts from legitimate system changes or updates. Refine rules and use ignore lists to reduce noise without missing real threats.
• Agent Management: Ensure agents remain updated and communicate reliably with the manager. Use automation tools for large-scale deployments.
• Compliance Alignment: Regularly review and tailor FIM rules to align with compliance requirements and evolving threats.
• Security of Communication: Always secure agent-to-manager communication using encryption and strong authentication, especially over untrusted networks.
• Do Not Use for Network IDS: This skill does not cover network-level intrusion detection or EDR capabilities-use specialized solutions for those scenarios.

By configuring host-based intrusion detection systems thoughtfully, organizations can significantly enhance endpoint visibility, protect critical assets, and meet stringent compliance mandates.