How to prevent malicious usage of VMware ESXi 4.1 in NTP reflection DDoS attacks
UPDATE: NTP in ESXi contains serious vulnerabilities and there is no official patches. I recommend to stop using ESXi altogether or disable the NTP service and accept the fact that the time will be inaccurate. December 24, 2014
Several days ago I've got a message from our hosting company. It stated that one of our IP addresses was used in massive UDP DoS attack. Since that IP address does not belong to any VM and is actually assigned to VMware ESXi host itself, I started my investigation. As it turned out:
- VMware ESXi 4.1 was installed by hosting company (probably with default settings)
- ESXi is based on Linux and contains ntpd 4.2.4p6
- By some unknown reason, default ntpd settings allow anyone to query daemon status using "monlist" command
- Monlist is a remote command that sends the requester a list of the last 600 clients who have connected to that NTP server
- NTP uses UDP as an underlying protocol, therefore source address can be easily spoofed
By scanning internet address space, attackers gathered a list of NTP servers which allow querying their status. Then these servers were flooded with NTP monlist command sent from spoofed source address. This spoofed address belongs to the victim being attacked. Each NTP server responds with a list of the last 600 clients which is significantly larger than original request (one 40-byte-long request generates 18252 bytes worth of response traffic). This leads to significant amount of UDP traffic which can be directed by attacker to any destination.
How to verify if your server is vulnerable
$ ntpdc -c monlist 1.2.3.4
remote address port local address count m ver rstr avgint lstint =============================================================================== example.com 64059 1.2.3.4 52 7 2 590 0 0 host.example.com 123 1.2.3.4 1 4 4 590 0 57 ....
How to fix
1. Get access to local or remote (ssh) ESXi console
Launch VMware vSphere Client and connect to your ESXi server. On the "Configuration" tab open "Security Profile" and start "Local Tech Support" or "Remote Tech Support (SSH)" service. To use local mode you need either physical server access or remote KVM. For remote mode, SSH client is sufficient.
2. Change NTP settings
Edit /etc/ntp.conf and add "noquery" to a line which contains "restrict default":
restrict default kod nomodify notrap noquery nopeer restrict 127.0.0.1 server pool.ntp.org driftfile /etc/ntp.drift
Do not forget to restart NTP daemon:
$ /etc/init.d/ntpd restart
And make sure it is running after restart:
$ ps ax | grep ntp
1234 1234 ntpd /sbin/ntpd
3. Verify that monlist command no longer works
$ ntpdc -c monlist 1.2.3.4
1.2.3.4: timed out, nothing received ***Request timed out
Other possible countermeasures
- Upgrade NTP to the latest version which do not contain "monlist" command (this is problematic on ESXi)
- Add ACL to inbound firewall or NTPD itself to allow only trusted clients
- Rate-limit responses to individual IP addresses
In an ideal world every provider should filter spoofed packets at their source (spoofed address does not belong to provider's networks). This is described in detail in BCP38
Related links
- Hackers Spend Christmas Break Launching Large Scale NTP-Reflection Attacks
- DoS attacks that took down big game sites abused Web’s time-sync protocol
- Technical Details Behind a 400Gbps NTP Amplification DDoS Attack
- NTP reflection attack
- NTP DoS reflection attacks
- Using Tech Support Mode in ESXi 4.1 and ESXi 5.x
- Editing configuration files in VMware ESXi and ESX
- Enabling remote query of NTP service on ESX/ESXi 4.0
Share this!
How to secure VMware ESXi host against being used in NTP amplification attacks (monlist command)?