How to prevent malicious usage of VMware ESXi 4.1 in NTP reflection DDoS attacks

UPDATE: NTP in ESXi contains serious vulnerabilities and there is no official patches. I recommend to stop using ESXi altogether or disable the NTP service and accept the fact that the time will be inaccurate. December 24, 2014

Several days ago I've got a message from our hosting company. It stated that one of our IP addresses was used in massive UDP DoS attack. Since that IP address does not belong to any VM and is actually assigned to VMware ESXi host itself, I started my investigation. As it turned out:

• VMware ESXi 4.1 was installed by hosting company (probably with default settings)
• ESXi is based on Linux and contains ntpd 4.2.4p6
• By some unknown reason, default ntpd settings allow anyone to query daemon status using "monlist" command
• Monlist is a remote command that sends the requester a list of the last 600 clients who have connected to that NTP server
• NTP uses UDP as an underlying protocol, therefore source address can be easily spoofed

By scanning internet address space, attackers gathered a list of NTP servers which allow querying their status. Then these servers were flooded with NTP monlist command sent from spoofed source address. This spoofed address belongs to the victim being attacked. Each NTP server responds with a list of the last 600 clients which is significantly larger than original request (one 40-byte-long request generates 18252 bytes worth of response traffic). This leads to significant amount of UDP traffic which can be directed by attacker to any destination.

How to verify if your server is vulnerable

$ ntpdc -c monlist 1.2.3.4

remote address          port local address      count m ver rstr avgint  lstint
===============================================================================
example.com            64059 1.2.3.4               52 7 2    590      0       0
host.example.com         123 1.2.3.4                1 4 4    590      0      57
....

How to fix

1. Get access to local or remote (ssh) ESXi console

Launch VMware vSphere Client and connect to your ESXi server. On the "Configuration" tab open "Security Profile" and start "Local Tech Support" or "Remote Tech Support (SSH)" service. To use local mode you need either physical server access or remote KVM. For remote mode, SSH client is sufficient.

2. Change NTP settings

Edit /etc/ntp.conf and add "noquery" to a line which contains "restrict default":

restrict default kod nomodify notrap noquery nopeer
restrict 127.0.0.1
server pool.ntp.org
driftfile /etc/ntp.drift

Do not forget to restart NTP daemon:

$ /etc/init.d/ntpd restart

And make sure it is running after restart:

$ ps ax | grep ntp

1234 1234 ntpd /sbin/ntpd

3. Verify that monlist command no longer works

$ ntpdc -c monlist 1.2.3.4

1.2.3.4: timed out, nothing received
***Request timed out

Other possible countermeasures

• Upgrade NTP to the latest version which do not contain "monlist" command (this is problematic on ESXi)
• Add ACL to inbound firewall or NTPD itself to allow only trusted clients
• Rate-limit responses to individual IP addresses

In an ideal world every provider should filter spoofed packets at their source (spoofed address does not belong to provider's networks). This is described in detail in BCP38

Related links

• Hackers Spend Christmas Break Launching Large Scale NTP-Reflection Attacks
• DoS attacks that took down big game sites abused Web’s time-sync protocol
• Technical Details Behind a 400Gbps NTP Amplification DDoS Attack
• NTP reflection attack
• NTP DoS reflection attacks
• Using Tech Support Mode in ESXi 4.1 and ESXi 5.x
• Editing configuration files in VMware ESXi and ESX
• Enabling remote query of NTP service on ESX/ESXi 4.0

Share this!

How to secure VMware ESXi host against being used in NTP amplification attacks (monlist command)?