linuxsecurityblog.com

Think there is a too much output with rkhunter? Including false positives on the binaries? If so this script should help.

(For the binaries, its good to check at least the md5 sum with something like AIDE).

1
2
3
4
5
6
7
8
9
10
11
12

#!/bin/bash
echo System checks summary >/home/user/rootkith.log
echo ===================== >>/home/user/rootkith.log
echo File properties checks... >>/home/user/rootkith.log
cat /var/log/rkhunter.log |grep Files\ checked >>/home/user/rootkith.log
cat /var/log/rkhunter.log |grep Suspect\ files >>/home/user/rootkith.log
echo Rootkit checks... >>/home/user/rootkith.log
cat /var/log/rkhunter.log |grep Rootkits\ checked >>/home/user/rootkith.log
cat /var/log/rkhunter.log |grep Possible\ rootkits >>/home/user/rootkith.log
echo Applications checks... >>/home/user/rootkith.log
cat /var/log/rkhunter.log |grep Applications\ checked |grep -v Info >>/home/user/rootkith.log
cat /var/log/rkhunter.log |grep Suspect\ applications >>/home/user/rootkith.log

Then you setup a cronjob to cat rootkith.log and mail you the output of it daily.

Here’s a command that can be helpful.

nc or netcat, what nc does is always you to write across the network.

For an example we will take #2 from my last post on how to handle a hacked hard drive:

#2. List the current open files, lsof, current processes, ps aux, current open ports netstat -anpe

Since you don’t want to write anything to the hacked drive, and you really want to get this output as fast as possible,   so an easy way to do that is to use netcat.  Ahead of the server getting hacked you will want to pick a port you will use for netcat, and open that for egress traffic on any server you might need to send out on, and ingress on one server you choose to be the netcat host to receive the data.  In this example we will use port 9999

First we need to start up netcat to listen for date, on the host run:

1

nc -l -n 9999 2>&1 | tee /dev/shm/netcat.tee

Then on the hacked server you want to send from run:

1

(lsof ; ps aux; netstat -anpe ) | nc ip_of_netcat_host 9999

Then you can less /dev/shm/netcat.tee and inspect the output from the hacked server.

One common misconception is when you get hacked you should reinstall the OS.

While its true you shouldn’t continue to run the OS, you definitely don’t want to immediately reinstall, and certainly not on that same drive.

Here is a quite write up I did for someone who said they got hacked on a forum, this isn’t the best case for every scenario, but is a lot better then what several people told him “reinstall the os”

#1. Do not install, reinstall or delete anything from that drive

#2. List the current open files, lsof, current processes, ps aux, current open ports netstat -anpe

#3. Pull the power cord out from the box (if possible or have the data center do it)

#4. Notify all your users that there has been a compromise, notify your provider if necessary.

#5. Make a forensic image of the drive (or have the data center do it) using the unix dd command, set the original drive in a safe place and ensure you maintain a chain of custody on it.

#6. Go through the logs you have from Chkrootkit / Rootkit Hunter / Aide / Samhain / Snort / Integrit / Osiris or tripwire, if the logs are on the drive itself look at them on the image your made.

#7. Review the image of the compromised drive, was the OS/kernel current? Were all the packages up to date? What was in the world writeable directories like /tmp, /var/tmp, /dev/shm, what services were running on the drive, what was the version of php, perl, etc.

#8. Look at the logs files and logrotated files such as wtmp, secure, messages, firewall logs setuid files, user shell histories, yum logs.

#9. Document any hints, hunches, or gut feeling you have on the the box was hacked.

#10. Only after your investigation and developing a plan to keep the box more secure should you install the OS on the new drive (the compromised drive should still be in a safe place) and only the user home data should be restore, and chowned to the user’s username, prior to the server being live on the internet again.

#11. Contact other parties, such as law enforcement if appropriate.

Here is a post on backups, which are a big part of security even though they don’t make the servers or network more “secure”.

Well, I bought 12 1.5TB drives (2 sets of 6) for my master backup server, however I’ve had a hell of a time getting it to work.

I tried both sets of the drives, 1-3 drives at time, accepting/reject the current configuration but with no luck.

The raid card does not pick up the drives, it does under the drive utilities section, but not under initialize drives or create array.

I even tried initializing the drives on another raid card and the bringing them back over, no luck.

Tried a new raid card, no luck.

I believe the problem is the raid card does not support 1.5TB drives and the limit is 1TB, which are the size of the old drives that were in there.

I hope I’m right because a new raid card is $580.

While I have this server unracked, I put an extra 2GB of ram in. rsync uses quite a bit of ram as it stores the file list in memory.

The raid card should be here Wednesday, and I’ll let you know if that worked.

TTYL,
DW

Secure Encrypted FormMail, PHP and GPG based.

We are all familiar with the original FormMail script created back in 1995, but are you familiar with a secure formmail script?

And by secure I mean one that encrypts the data.

I sure wasn’t so one night while I was up late, waiting for my scheduled maintenance window to come around, I typed one up.

On common misconception that I see a lot of people do is put an https URL for their formmail action tag, and then think the form is secure.

Well, that doesn’t help a whole lot, it would secure the data back to the server, but then emailing it to you would not necessarily be secure, especially if your mail server is different than your web server, plus there will be a period of time where the form results are in plain text on the server.

What you need is not only a formmail that will use https, but that will also encrypt the data with GPG.

That said I bring you Secure Formmail:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186

<?php                                         
//Doug Walker's Secure Formmail Copyright 2009 FreeGPG.org
//This script is provided with absolutely no guarantee or warrrenty
//This script is not assumed to be bug free or error free          
//Secure Formmail may not be 100% secure, and may not be secure enough or appropriate for your specific use
//We are not responsible for misencrypted or unencryptable messages or damages caused by use of this script
//You agree to limit any loss from the use of this script to $5, which you agree is a fair amount since the script is free
//This script was developed using GnuPG v1.4.5                                                                            
//This script is meant to be used in conjunction with SSL (https://)                                                      
 
//***** YOU MUST SET $referrers $recipients AND ENTER YOUR PUBLIC KEY*****
 
//Set valid referring URLs, seperate multiple approved referrers with a , no spaces
$referrers="domain.com,www.domain.com";                              
 
//Set valid recipient domains
$recipients=$referrers."emaildomain.com,us.emaildomain.com";
 
//include your publickey, by default it is in publickey.txt in the same directory as this script
//include the Begin and End lines and no extra spaces                                           
//You can get a public key from freegpg.org but are not required to use that public key         
//***Paste the your key starting with the BEGIN line right below <<<EOI                         
//***and with your END line right above EOI;                                                    
$publickey=<<<EOI                                                                               
-----BEGIN PGP PUBLIC KEY BLOCK-----                                                            
Version: GnuPG v1.4.5 (GNU/Linux)                                                               
 
qwerwqerwer
YOUR PUBLIC KEY HERE
qwerqwerwq                                                      
-----END PGP PUBLIC KEY BLOCK-----                              
EOI;                                                            
 
// Escape the key and message for added security
$publickey=escapeshellarg(trim($publickey));    
 
//Declare variables
$secure=1; //Set secure mode 0 = off 1 = on
define('GPG', '/usr/bin/gpg');  // The gpg binary
define('HOME', '/tmp');         // .gnupg will be created here
putenv('HOME=' . HOME); //Set the home environment            
$to_max_len=200;                                              
$ref = getenv("HTTP_REFERER");                                
$timestamp = date("m/d/y  H:i:s", time());                    
$ip = $REMOTE_ADDR;                                           
$msg="";                                                      
//set referrer                                                
if($_SERVER['HTTP_HOST']!="")                                 
  $referrer=$_SERVER['HTTP_HOST'];                            
else                                                          
  $referrer=$_SERVER['HTTPS_HOST'];                           
//end set referrer                                            
$recipient=$_POST['recipient'];                               
 
//Begin the message with the referring URL, date/time stamp, and remote IP
$msg.="$ref\n$timestamp\n$ip\n\n";                                        
//$msg.="$ref<br>$timestamp<br>$ip<br><br>";                              
 
//validate referring URL
//initialize switch     
$sw=0;                  
$ref_array = explode(",", $referrers);
foreach ($ref_array as $ref_ele){     
  if($ref_ele==$referrer)             
    $sw=1;                            
}                                     
if(!$sw)                              
  die("Error: Invalid referring domain");
 
 
//check reciepient is set
if($recipient=="")       
  die("Error: No recipient set");
 
//check recipient length
if(strlen($recipient)>200)
  die("Error: Recipient length too long");
 
//validate recipient
//get domain portion of recipient
$recip_email_arr=explode('@',$recipient);
$recip_dom=$recip_email_arr[1];          
 
//set from email
$from_email="webmaster@".$recip_dom;
 
//initialize switch
$sw=0;             
$rec_array = explode(",", $recipients);
foreach ($rec_array as $rec_ele){      
  if($rec_ele==$recip_dom)             
    $sw=1;                             
}                                      
if(!$sw)                               
  die("Error: Invalid recipient domain");
 
 
//The Guts
//required field error switch
$err_sw=0;                   
$err_msg="The following field(s) were left blank, please hit back and fill in:<br>";
 
foreach ($_POST as $field => $formvar){
  //switch to add to the message or not
  $sw=1;                               
 
  //Check if required field
  if($required!=""){       
    //build required field array
    $req_arr=explode(",",$required);
    foreach ($req_arr as $req_ele){ 
      if($req_ele==$field)          
        if($formvar==""){           
          $err_sw=1;                
          $err_msg.="* $field<br>"; 
        }                           
    }                               
  }                                 
 
 
  //Check for recipient
  if($field=="recipient")
    $sw=0;               
 
  //Check for subject
  if($field=="subject")
    $sw=0;             
 
  //Check for requied post var
  if($field=="required")      
    $sw=0;                    
 
 
  //Check for redirect URL
  if($field=="redirect")  
    $sw=0;                
 
  //for testing only
  #echo "$field: $formvar<br>";

  //Build the message
  if($sw)
    $msg.="$field: $formvar\n";
}
 
//display error message if required fields were left blank
if($err_sw)
  die($err_msg);
 
//Encrypt the message
if($secure){
 
  // Import the key into the keyring
  $result=shell_exec("/bin/echo $publickey | " . GPG . " --dearmor | " . GPG . " --import - 2>&1");
  // Find the key ID so that we can remove it later
  preg_match('/key ([A-Z0-9]+):/', $result, $matches);
  if (empty($matches[1])) {
   die('Problem importing key!');
  }
  $key_id=$matches[1];
 
  // Escape the key and message for added security
  $message=escapeshellarg(trim($msg));
 
  // Fire up GPG and encrypt the message
  $result=shell_exec("/bin/echo $message | " . GPG . " --batch --recipient '$key_id' --trust-model always --armor --encrypt 2>&1");
 
  $msg=$result;
 
  // Remove the key from the keyring
  $result=shell_exec(GPG . " --batch --yes --delete-key $key_id' 2>&1");
  if (!empty($result)) {
    die('Error: 29391');
  }
 
}
 
//Send the email
mail($recipient,$subject,$msg,"From: $from_email");
 
//Redirect the user if redirect set
if($redirect!="")
  header( "Location: $redirect" );
else
  header( "Location: http://$referrer" );
?>

The nice thing about this script is, its easy to use. Simply paste in your public key, and if you don’t have one, get one at freegpg.org, and then set the referrers URL, and if necessary change the path to gpg. This script has been tested and works well with the shared web hosting servers of Penguin Web Hosting

After the form is submitted, it sends you an email with the GPG encrypted message, and then you can decrypt it with your gpg compatible email client, or paste it in, along with your private key, at FreeGPG.org.

Secure FormMail can accept the 4 common hidden fields:

1
2
3
4

<input type=hidden name=recipient value="">
<input type=hidden name=required value="">
<input type=hidden name=redirect value="">
<input type=hidden name=subject value="">

Hope you find this useful!

TTYL,
Doug Walker

Got GPG keys?

If not, you can get them at https://www.freegpg.org

This is an online interface for:

• creating your own public and private key (the private key is not stored online)
• encrypting and decrypting messages
• looking up the public key of other freegpg members

The online interface is a lot easier than the plugins for the mail clients, especially for windows, plus the look up feature is nice, because I never remember where I put everyones public key, or I reformat and the don’t have them any more, or the person has to generate new keys, and I encrypt a message with their old public key.

Here is my public key (dougw[at]linuxsecurityblog.com):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.5 (GNU/Linux)
 
mQGiBErzpG8RBACOKTmbCjMlerI91DWWRFEMntIAG7SJ4DSAdVF7eocuVbFF/c3F
LESCkrP+sjmLy5+B3A1rn3XoSjRgLQ3OxBA/cc93Yu3YbTBAfYEY7u1pvRGEqS+F
cOrfYoK3QI3Gd536tgE8xtiV9ry5ROIYx/O67URdrXB1WKhh7X4qdbJLVwCg8pvn
Jc8/4RplgZIGNAJESEN5NcsD/A4wM940yAKoFqCEHdC3goqakONuEDkQE9w2UWcj
8Zex4xMkyWOX0ZpkuWYytVWipSH9r9gQvJSa4dQppf4z98P2lOFOAKYhy7QNI/M2
KJuG1DAHLKfaDOkwi2fU4J4DAtq52oIaE+K13FSFi8vILl0GXl84TQ+DpbMpGIWS
+j93A/4+gSJyn35p8Dq3i1BRH3k9U2Eb+qqKx0uxG4XkCug8yasMe+IIDA9UIKcq
7OCtSDLp5fe2G1DTinBstrJ1ri9Vwn93qxcN7HWEUdqszw2pzHe+GmCAdv5D583R
m06IPve4JIGKymuNaIHi4cGqCYNig1VfcMP/M7wxjlYIGUPIWrQnR1BHIENyeXB0
IDxkb3Vnd0BsaW51eHNlY3VyaXR5YmxvZy5jb20+iGAEExECACAFAkrzpG8CGyMG
CwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRB98qmyKko4wyZiAJ9nDeGs2tPr/JWC
igFkKXCLcAC2/gCg36p2XjfbRAR1mz5PjrSlaMcIgi+5AQ0ESvOkbxAEALPVXri9
0EbZZMkxJh41WctRTqvy2cKQwTr8aVqhbmNQhBN2BD00OP/QbokNc9rpW4Z4SThV
LRXPAk7dwSxDQfZnMuGbg9V6vgXL5sdZwKh8t4iUdoR5jSe0SPWnaTuAXsV50NL5
nrmrMWonXT51qjWn/dZlsQxXWYyo/+ewZOoHAAMFA/9TeC+mk6ShcEoYXNh5SVp0
cFP/bl5MaUrPBbWbtg7SrJF9KZX/ZhoO3eh2wbtAj//FNmWSVmi8UYsh50MeXqVF
kI0T1JIIbLUxZ3WtPfbKMRHMpPj915EsiooJ4vvjnZ8kPCsWe2QWiavtYfOTKj/b
5esn8vlIXsaGAXHSxdYbb4hJBBgRAgAJBQJK86RvAhsMAAoJEH3yqbIqSjjDltAA
oOdRJ0OG81rV6MX60GrD6JZfzyMFAKCCWmt4sRwlVA1qrXoh9gadGRpezw==
=Un5z
-----END PGP PUBLIC KEY BLOCK-----

And for those if you wondering, yes, I’m the founder of freegpg.org

TTYL,
Doug Walker

Howdy All,

I installed a new code formatting plugin, so I will try to give some code examples as often as possible in my posts.

I was working on a hacked site yesterday, and there was a file -tmp that couldn’t be removed with a regular plain rm.

Linux allows you to create a file that starts with a – but when you try to remove it that’s a little more tricky.

Creating a file that starts with a -

1
2
3

$ >-test
$ ls |grep test
-test

Now if you try to remove that file:

1
2
3
4

$ rm -test
rm: invalid option -- t
Try `rm ./-test' to remove the file `-test'.
Try `rm --help' for more information.

Now to actually remove a file that starts with a -, there are actually two ways:

1
2
3
4
5

$ rm ./-test
$ ls |grep test
$ >-test
$ rm -- -test
$ ls |grep test

TTYL,
Doug Walker

Well yesterday was a busy day.

Among some Joomla upgrades, a form that sends an encrypted email, and a couple sites infected with malware, on of the malware sites, while I was working, and as soon as I would remove malware it would reappear.  I ended up locking down the site, while I removed everything.  Looking at the FTP log, I would see many different IPs trying to connect to it.  After resetting his password I noticed failed logins for his account from almost 400 different IP’s from all over the world.  I suspect this was a botnet set to continually infect his pages once they noticed they code was removed.

After getting him all cleaned up, aide has detected no file changes in about the last 24 hours.

And this morning, I have some more malware to remove from another site.

TTYL,
Doug Walker

Happy Halloween folks!

My new computers is finally finished and how appropriate for Halloween, are the 2 skull fan grills with black lights.

New Computer

This took about 2 months working on it when I had some spare time.

Features include:

• Blue acrylic case
• Red cold cathodes
• Black lights
• Skull fan covers with black lights
• See through power supply
• 4 led switches
• 4 sets of bright leds
• UV reactive wires, sata cables, heat shrink, and molex’s

For the hardware I went with:

• Asus Motherboard
• Crucial Ram with leds (4GB)
• AMD CPU
• 3 1TB harddrives
• Raid 1 (hardware raid)

And my Linux OS of choice for this was Kubuntu.

I’ve done a few mods before, but this was the most complicated one, as I have not wired switches before, and instead of daisy chaining a bunch of molex connectors I soldered a lot of the wires together.

Naturally I ran into a problem, I wanted to run the lighted side fan grills through a switch, along with the fans themselves, but the grills are 5V and the fans are 12V, I realized this when I went to plug it into the switch harness and said wait a minute, there’s only two wires on the harness.  Then a light came on, and I realized I can’t run 2 different voltages through a single pole single throw switch. So those are now on all the time, and I will either get a different switch, or add something else to plug into the empty switch.

TTYL,
Doug Walker

Well, after tell my clients to upgrade their scripts yesterday, I’ve been busy fielding questions, and upgrading the people who don’t know how.

One guy in particular is already infected with malware.

This is what I just told him:

Your site is already infected with malware, we are going to try to remove that for you.

I disabled these two old joomla installs:
public_html/joomla/tmp/install_48698a1f8c122/libraries/joomla
public_html/joomla/tmp/install_4869bd8b81e25/libraries/joomla

They are in tmp directory, did you want me to remove them?

Here is part of the malware code, which is on your index.html, I’m going to scan you other pages for it:
<script type=”text/javascript”>var iquXiQSJiqVYjfLaNHOA = 99UNtj105UNtj116UNtj121UNtj61UNtj4Ntj60UNtj

And sure enough he has other pages infected with the malware.

I’m now grepping his other pages for the malware, and what I have noticed on other sites, is the hacker will vary the code, so I have to grep based on a regex as a text string will find only the one page.

And here is some info to support my earlier claim that the average number of pages hacked in a site is greater than 10:

# grep -R  un*******ant * |wc -l
457

Well, I am off to formulate a quick command to remove those.

TTYL-
Doug Walker