Monitoring The Kernel
- What modules is the Kernel using (this we can monitor and look at).
- Whether hardware has been detected.
- The Kernel cannot read the hardware properly.
- Performance is under control of the Kernel as well.
- Another Operating System called
Plan 9, originally from AT&T’s Bell Labs- Wanted to represent everything as a
file. memory, network communications etc as files.
- Wanted to represent everything as a
- Checking
/dev, all of the files there are representations of the hardware. - The Kernel represents a lot of what it is doing as files as well.
- We can check this in
/proc - If we back up to
root, we can see that some are symlinks (points to other directories). - If you check the permissions for
proc, these look a little strange:dr-xr-xr-x 337 root root 0 Jan 24 08:52 proc
- We can check this in
- The
337above represents the amount of links to that directory.- Only
roothas read and execute and no one has write permissions. procis a pseudo directory and not a main directory.- Representing the activity of the Kernel.
- All of the files inside
/proccontain data on what the Kernel is currently doing. - To find out the version of the Kernel we are running, we go into
/proc–>/proc/sys/kernel- In that directory, you will find a file called
version - If you then
cat version, you then receive an output like this:howard@explosion:~$ cat /proc/sys/kernel/version #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023
- In that directory, you will find a file called
- Only
- When running
catand receiving the output of the above, we interacted with the Linux API. - When you run a program, you are not actually running it - you are asking the Kernel to run it for you.
- For example editing a file with
vim - While
vimis open, we find thevimprocess withps aux | grep vimYou wil see the process ID outputted.- Back in the
/procdirectory, you can see all of the running processes, we cancdinto the process ID directory and see exactly whatvimis doing.- If you then run
ls -linside the process directory and thengrepforcwd(current working directory), it symlinks back to theuser'shome directory (where hte command is being ran from). - The symlink
exealso shows the binary that was ran, for exampleexe -> /usr/bin/vim.basic - We can also check the file
cmdline, which shows the exact file that was ran:vim file1.txt - We can view all of the environmental variables that the application is relying on as well.
- We do this with
cat environand it shows all of the variables that are in use.
- We do this with
- If you then run
- Back in the
- For example editing a file with
- We can also shorten the commands we are running, for example
cat /proc/sys/kernel/versioncan output the same as if we had doneuname -vuname -vsays “give me the Unix Kernel Name and Version”. As an example:#101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023
- The #101 is a tile.
- Again, using
catdoesn’t access the file directly and it goes via the Linux API to talk to the Kernel.
- Again, using
- We can also do
uname -aand it will collect information from multiple places and represent it back to us. - We can then do
cat /proc/uptime- Shows the time in minutes and seconds.
uptimecommand presents the exact same information, but in a human readable format.
- To the see the modules that have been loaded, we can do
cat /proc/modules- Of course
lsmoddoes the exact same thing and is formatted much more nicely.
- Of course
- However, the
/procdirectory is useful, if you do not have commands avialable such aslsmodon a particular system. - It is possible to write to several locations within
/proc, it is not allread-only. It is possible to change the Kernel’s behaviour.- Can implement changes from
/proc/sys/kernel - Some scenarios where this has to happen.
- Google has an outage in their GCP. Took 3.5 hours to come back online. Google was re-provisioning some servers. During the re-provisioning, they had a shared caching database. When the servers tried to connect to each other, they exceeded the maximum amount of open connections that the Linux Kernel allows by default. The Kernel has limits that are put in place. The engineers needed to increase the limit within the Linux Kernel.
- An example is the maximum amount of open files. This would be
/proc/sys/fsfor file system.- Inside, you will see
file-max, which is the maximum amount of files you can have open. file-nris the amount of files that I currently have open. For example we havehoward@explosion:~$ cat /proc/sys/fs/file-nr 12448 0 9223372036854775807and so12448are currently open.- If we want to change
file-max, we can just open that as a text file and then save it. If you make a typo, that can cause a lot of issues.- The utility used instead is
sysctlor System Control.sysctlis designed to interact with the Kernel and set values accordingly.- We then do
sudo sysctl fs.file-maxWe want to set the filesystem’s maximum files. This will return a value in the style offs.file-max = 1000000 - We can then write with
sudo sysctl -w fs.file-max=2000000- If you reboot however, this doesn’t persist and it goes away.
- If you want the changes to persist, you have to store them in the filesystem.
- In older Linux distros, you can change
/etc/sysctl.conf, make the changes to the Kernel there and they then persist. There is a file afterwards calledsysctl.d. Example ofsysctl.dfor Linux Mint:howard@explosion:/etc/sysctl.d$ ls 10-console-messages.conf 10-ipv6-privacy.conf 10-kernel-hardening.conf 10-magic-sysrq.conf 10-network-security.conf 10-ptrace.conf 10-zeropage.conf 99-sysctl.conf README.sysctl
- In older Linux distros, you can change
- If you want the changes to persist, you have to store them in the filesystem.
- If you reboot however, this doesn’t persist and it goes away.
- We then do
- The utility used instead is
- Inside, you will see
- An example is the maximum amount of open files. This would be
- Google has an outage in their GCP. Took 3.5 hours to come back online. Google was re-provisioning some servers. During the re-provisioning, they had a shared caching database. When the servers tried to connect to each other, they exceeded the maximum amount of open connections that the Linux Kernel allows by default. The Kernel has limits that are put in place. The engineers needed to increase the limit within the Linux Kernel.
- Can implement changes from
- Each of the above changes are numbered at the start and are applied in order from lowest to highest.
- To add your own, you do
sudoedit 00-custom-settings.conf. Example contents would be:fs.file-max=2000000 - Would need to reboot the machine to take effect.
- To process any changes WITHOUT a reboot, we do
sudo sysctl -pto process any changes.
- To process any changes WITHOUT a reboot, we do