Useful Commands For Checking IO and Networking-related Issues

The command:

auditctl -a always,exit -F arch=b64 -S read -k fs_read

is used to configure the Linux Audit subsystem to monitor specific system calls. Here’s a breakdown of what each part does:

🔍 Command Breakdown

auditctl: This is the command-line utility to configure the audit system in real time.
-a always,exit: This tells the audit system to always audit the exit point of a system call. That means it logs the result after the system call has been executed.
-F arch=b64: This filter specifies the architecture of the system call. b64 means 64-bit architecture. On a 64-bit system, you may also want to add a similar rule for b32 if you want to catch 32-bit syscalls too.
-S read: This specifies the system call to monitor. In this case, it’s the read() syscall, which is used to read data from a file descriptor.
-k fs_read: This assigns a filter key (fs_read) to the rule. This key is useful for searching logs later using ausearch -k fs_read.

✅ What It Does

This rule logs every time a 64-bit process calls the read() system call, which is typically used to read from files, sockets, or other file descriptors. It’s useful for tracking file access or data exfiltration attempts.

🧪 Example Usage

To view logs generated by this rule:

ausearch -k fs_read

This command:

inotifywait -m -r /overlay/mount/point -e open,read,access > /var/log/overlayfs_reads.log &

is using inotifywait to monitor file system activity. Here’s a detailed breakdown of what it’s doing:

🔍 Command Breakdown

inotifywait: A command-line utility from the inotify-tools package that waits for changes to files using the Linux inotify API.
-m: Monitor mode — keeps running and doesn’t exit after the first event. It continuously watches for events.
-r: Recursive — watches all subdirectories under /overlay/mount/point.
/overlay/mount/point: The target directory being monitored. This is likely the mount point of an OverlayFS filesystem.
-e open,read,access: Specifies the events to watch for:
- open: A file was opened.
- read: A file was read (note: not all versions of inotifywait support read; access is more commonly used).
- access: A file was accessed (read or executed).
> /var/log/overlayfs_reads.log: Redirects the output (event logs) to a file for logging purposes.
&: Runs the command in the background, allowing the shell to continue accepting other commands.

✅ What It Does

This command sets up a background process that continuously monitors the /overlay/mount/point directory and all its subdirectories for any file access, open, or read events. When such an event occurs, it logs the details to:

/var/log/overlayfs_reads.log

This is useful for auditing file access, debugging, or detecting unauthorised reads in environments using OverlayFS (e.g., containers, live systems, or layered filesystems).

The command:

xfs_repair -n /dev/sdX

is used to check the integrity of an XFS filesystem without making any changes. Here’s a breakdown of what each part does:

🔍 Command Breakdown

xfs_repair: This is the utility used to repair corrupt or inconsistent XFS filesystems.
-n: This is the no-modify or dry-run option. It tells xfs_repair to only check the filesystem and report problems, but not fix them. It’s useful for safely assessing the health of the filesystem.
/dev/sdX: This is the block device representing the XFS filesystem you want to check. Replace sdX with the actual device name (e.g., /dev/sda1).

✅ What It Does

Scans the XFS filesystem on /dev/sdX.
Reports any metadata inconsistencies or corruption.
Does not modify the filesystem — it’s safe to run on a mounted filesystem.

⚠️ Important Notes

If you want to actually repair the filesystem, you must unmount it first and run:
```
xfs_repair /dev/sdX
```
Running xfs_repair without -n on a mounted filesystem can cause data corruption.

Would you like help interpreting the output of xfs_repair -n, or guidance on safely repairing a filesystem?

The command:

iostat -xz 5 > /var/log/iostat.log &

is used to monitor system I/O performance and log it continuously. Here’s a detailed breakdown:

🔍 Command Breakdown

iostat: A utility from the sysstat package that reports CPU statistics and input/output statistics for devices and partitions.
-x: Shows extended statistics for each device (e.g., utilisation, queue size, wait time).
-z: Suppresses output for devices with no activity, making the output cleaner and more focused.
5: Tells iostat to repeat the report every 5 seconds.
> /var/log/iostat.log: Redirects the output to a log file, overwriting it each time the command is restarted.
&: Runs the command in the background, so your terminal remains usable.

✅ What It Does

This command continuously logs detailed I/O statistics every 5 seconds into /var/log/iostat.log. It’s useful for:

Diagnosing disk bottlenecks
Monitoring disk performance over time
Collecting data for later analysis

📌 Example Output (simplified)

Device:    rrqm/s wrqm/s r/s w/s  r_await w_await svctm %util
sda          0.00   0.00 1.00 2.00   1.50    2.00   0.50  10.00

This shows read/write rates, wait times, service times, and utilisation for each device.

The command:

blktrace -d /dev/sdX -o - | blkparse -i - > /var/log/blktrace.log &

is used to trace and analyze block I/O operations on a specific device in real time. Here’s a detailed breakdown:

🔍 Command Breakdown

blktrace: A tool that captures detailed I/O events from the block layer of the Linux kernel.
-d /dev/sdX: Specifies the device to trace (replace sdX with your actual device, e.g., sda).
-o -: Directs the output to stdout (standard output) instead of a file.
| blkparse -i -: Pipes the output of blktrace into blkparse, which parses and formats the raw trace data into human-readable form.
> /var/log/blktrace.log: Redirects the parsed output to a log file.
&: Runs the entire pipeline in the background.

✅ What It Does

This command starts a real-time trace of all block I/O operations on /dev/sdX, parses the trace data, and logs it to /var/log/blktrace.log. It captures events like:

Queueing
Merging
Dispatching
Completion of I/O requests

This is extremely useful for:

Performance tuning
Debugging I/O bottlenecks
Understanding how the kernel handles disk I/O

📌 Example Use Case

If you’re experiencing high disk latency or want to analyze how a specific workload interacts with the disk subsystem, this tool gives you low-level visibility into those operations.

The command:

journalctl -k -f > /var/log/dmesg_live.log &

is used to continuously monitor and log kernel messages in real time. Here’s a breakdown of what each part does:

🔍 Command Breakdown

journalctl: A command-line tool to query and display logs from systemd-journald.
-k: Filters the output to show only kernel messages (equivalent to what you’d see in dmesg).
-f: Follows the log output in real time, similar to tail -f.
> /var/log/dmesg_live.log: Redirects the real-time kernel log output to a file, overwriting it each time the command is restarted.
&: Runs the command in the background, so your shell remains usable.

✅ What It Does

This command starts a background process that:

Continuously watches for new kernel messages (e.g., hardware events, driver logs, kernel warnings).
Writes them in real time to /var/log/dmesg_live.log.

This is useful for:

Live monitoring of kernel activity
Debugging hardware or driver issues
Capturing logs during system events like device hotplugging or kernel panics