Securing Linux 101: Reasonable Steps to Detect and Prevent Blackhats

From unixvreview.com by Jeff Kopmanis

In the age of Web-based archives of packaged break-in tools (“script kits”) being run by inexperienced hackers (“script kiddies”), it’s increasingly important not to overreact when you suspect that a break-in has occurred. This article helps provide ideas, methods, and checklists for detecting blackhats and securing your Linux box. The article also provides a list of resources.

Blackhat Detection

System Security and Squirrels

“I'm running GreenBeret Linux 9.2.5c, the secure, commercial version. I'm not vulnerable!” Gimme a second to stop laughing.

• Anecdotal Lesson #1: Security is only relative. If your machine touches the Internet for any length of time, you’re vulnerable. If you don’t believe it, you’re fooling yourself and setting yourself and your organization up for a security snafu. Blackhats are very clever and, much like squirrels breaking into bird feeders, will try again and again until they succeed. Ever noticed that most squirrels are curiously plump? I just read a paper on GETSLAG from SensePost (http://www.sensepost.com) that outlines a technique to use a Windows browser to transfer data in and out over HTTP, or to potentially run other programs with obvious sinister consequences.

Curbing Your Instincts

Proper defenses can be erected only if you know something about how the break-in occurred, which tools have been compromised, and where your system fits into the blackhat’s scheme. If you suspect that you’ve been cracked, its critical to collect information about the event to prevent a repeat performance. Although yanking out the Ethernet cable and doing a hasty shutdown will instantly secure your system and make you feel like you’re back in control, it actually does nothing to prevent a blackhat from returning once your system goes back online. I once yanked the Ethernet out of annoyance, and spent the next hour reinstalling and securing the system, only to find that within 60 seconds of plugging the system back in, the cracker had blasted my work.

• Anecdotal Lesson #2: The blackhat already knows how you think, so to beat him, you need to figure out how he thinks. To do this, you must overcome your reaction to “squash that intruder, right now!” To beat a cracker, you need to spend some time investigating what he or she has done. The investigation necessary to keep the blackhat out is different from the normal sys admin-type of problems. In systems administration, often you want to simply get the user back up and running as soon as possible, minimizing the downtime. Stopping a blackhat is different in that it's now mandatory to understand what he did and how to stop him from a repeat performance.

Getting Started: Symptoms of a Break-In

Here are some common symptoms of abnormal system behavior:

• System load is higher than you expect.
• Your network activities are slow, take a long time to connect, or even disconnect.
• The hard disks are going nuts during seemingly idle(?) times.
• You’re using more RAM or swap space than you’d thought.
• Hey! I didn’t notice that program running before…
• Hey! Where did my /var/log directory go?
• Hmmm…netstat (or ps, or top, or df) just core dumped…
• Hmmm…I wonder what /dev/.uuu might contain?

Strangely enough, your system probably won’t crash.

• Anecdotal Lesson #3: It's not in the interests of the blackhat to deny himself your system’s resources. He’s interested in denying you your system’s resources. You’re looking for stuff that isn’t normal, which is part of the reason I advocate package managers — they make it trivial to check what’s changed in your install.

To begin, we'll log in as root and start looking for those pesky rodents!

Verifying Your Toolbox

For these examples, I'll mention a number of system utilities that can be used to snoop around your system looking for signs of a blackhat. One of the common things that blackhats do is to break or erase the very tools you’ll need in order to find them. Here’s a list of tools, files, and daemons that are often affected (ordered by importance):

• File utilities: rpm, ls, mount, umount, less, more, df, du, passwd
• Process Status utilities: ps, top
• Network status utilities: netstat, ifconfig

There are lots more, but most investigations only need the above. I suggest keeping a floppy disk or CD containing valid copies of these utilities, so that when you suspect trouble, you can mount the respective disk and use those utilities instead of trying to get verified copies of them to run.

How Do You Verify These Things?

With RPM, life is good because this tool can be used to check the database. You can check every file on your machine with the following command (I piped it into less so you could scroll back):

rpm -Va | less

Here’s some of what you’ll see in the above command:


S.5....T c /etc/sendmail.cf
S.5....T c /etc/ldap.conf
S.5....T c /etc/httpd/conf/httpd.conf
S.5....T c /etc/logrotate.d/apache
......G. /usr/sbin/suexec
.....U.. /var/cache/httpd
S.5....T c /etc/amd.conf
S.5....T c /etc/amd.net
S.5....T c /etc/my.cnf
..5....T c /etc/mime.types
S.5....T c /etc/logrotate.conf
What you’ll see is a field of flags (see below) and the complete pathname and filename of the files checked. For files you know you’ve modified, you don’t really need to worry about differences from the norm because you know you’ve modified them. Even so, it might still be worth a check. The flags below will help you establish the state of the files in question. From the rpm man page: “man rpm”:

S file Size differs
M Mode differs (includes permissions and file type)
5 MD5 sum differs
D Device major/minor number mis-match
L readLink(2) path mis-match
U User ownership differs
G Group ownership differs
T mTime differs

If you start to see one or more of the utilities above with MD5 or size differences, then you either didn’t upgrade with RPMs, or you’ve been hacked.

Reinstalling Your Toolbox

If the utilities are compromised, you must first find out what packages they came from in order to locate the proper packages on the disk. Here’s a quick script (name it “check” and do a chmod 700 check) to produce a listing of packages needed to install to replace that stuff:


#!/bin/tcsh
set utils=";rpm ls mount umount less more df du \
ps top passwd netstat ifconfig";
foreach u (`echo $utils`)
set f='which $u'
echo 'rpm –qf $f` >> /root/check.check
end
sort –u check.check > /root/check.out
/bin/rm /root/check.check:
Now, you must mount the distribution CD and reinstall things such as:

cd /mnt/cdrom/redhat/rpms
rpm --force -i 'cat /root/check.out'

You now have a solid base from which to check things out. Next comes the fun part!

Who Are You?

Your account information and your password are stored in the /etc/passwd and /etc/shadow files, respectively. The UNIX group definitions are stored in /etc/group. Together, they provide the basic authentication for your system (unless you’re required to use a scheme like Kerberos for authentication and ticketing). These files are also “classic” places for blackhats to meddle. I recommend checking them out before proceeding.

• Check that the file dates on /etc/passwd and /etc/shadow are reasonable.

• Use wc –l /etc/passwd /etc/shadow to verify that both files contain the same number of lines (users).

• Use less /etc/passwd to look for new, unauthorized users. If you find one, get out of less with the q command, and run vipw and edit out the users that aren’t supposed to be there. Note that vipw is the “proper” editor to use, because it locks the /etc/passwd file from changes while you’re editing. Vipw (and vigr below) are special versions of the venerable UNIX editor, vi.

Security Hole: As of this writing, vipw (version 1.4) has a bug in it that changes the permissions of the /etc/shadow file to 644 (rw-r-r-- ). After running vipw, you should issue the following chmod command to fix this:

chmod 600 /etc/shadow

(If you’re not a vi user, become one. vi is the only editor that is on every UNIX system. If you don’t know vi, the dd command will delete the current line, and the :wq! command will save and quit.)

• Use the above to find service accounts that have been given valid programs to log into. At the very end of each line in the /etc/passwd file, you’ll see something like /sbin/nologin, or /dev/null, or /bin/false. If you have an account called "nobody" with a login shell of /bin/tcsh, this is pretty suspicious.

• If the above steps reveal no glaring problems, it's safe to change your root password and any other service accounts you may have created. It's suggested to do that immediately:

passwd root

• Check the /etc/group file similar to the /etc/passwd. Use vigr to edit the group file.

Note that once you delete unauthorized users and groups, any programs running under those IDs will continue to run, and you’ve merely prevented the blackhat from logging in under that user again. In this situation, it's important to delete the unauthorized users before killing their blackhat activities and alerting them that you’ve found out. Frankly, killing off those processes is the very last thing I do. I want to be in a good position not to let them in once I’ve made it known that I’m aware of their activities by killing off a process. They may figure out that you’re working to keep them out before you kill them, but why tip your hand earlier than you have to?

• Anecdotal Lesson #4: Let your blackhat think he’s outsmarted you until you’re sure you’re the one doing the outsmarting.

Netstat: King of All Blackhat Detectors

You must become close buddies with the netstat utility. It's by far the best utility for getting a clear picture of what the fat squirrels are doing on your system. The Most Popular Command Option Combination Award goes to the -atu options that show you all of the tcp and udp (i.e., “Internet”) sockets that are open. netstat -atu will generate something like this (taken from a very quiet little system):


[root@dev root]# netstat -atu
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:32768 *:* LISTEN
tcp 0 0 *:mysql *:* LISTEN
tcp 0 0 *:netbios-ssn *:* LISTEN
tcp 0 0 *:sunrpc *:* LISTEN
tcp 0 0 *:http *:* LISTEN
tcp 0 0 *:ftp *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 localhost.localdom:smtp *:* LISTEN
tcp 0 0 localhost.localdo:32794 *:* LISTEN
tcp 0 0 *:https *:* LISTEN
tcp 0 272 dev.localdomain:ssh jeffk.localdomain:49226 ESTABLISHED
udp 0 0 *:32768 *:*
udp 0 0 dev.localdom:netbios-ns *:*
udp 0 0 *:netbios-ns *:*
udp 0 0 dev.localdo:netbios-dgm *:*
udp 0 0 *:netbios-dgm *:*
udp 0 0 *:sunrpc *:*
udp 0 0 *:893 *:*

It’s pretty easy to see that a MySQL database is running and that daemons are running that cover FTP, HTTP, HTTPS, SMTP, and SSH protocols. Not so obviously, Samba/SMB (netbios-ssn, netbios-ns, netbios-dgm) is running and some kind of RPC daemon (sunrpc) or support is available. It's also painfully obvious that I was connected via ssh to dev.localdomain from jeffk.localdomain (the “ESTABLISHED”) line.

Here’s the important part — look at a listing like this when your system is quiet. Pipe it into a sample file if you don’t remember it. When your system is under attack, and when you’ve got a good copy of netstat running, check the listing again. You’re looking for lines that show established or listening sockets on ports that you don’t know about. LISTEN means that a program is sitting around waiting for activity from the outside to activate it. ESTABLISHED means that a connection has already been made and the program (a blackhat?) is active and communicating to the network.

In the /etc/services file, there is a “registry” of sorts of TCP and UDP services. The reason that ssh or mysql show up as text and not as numbers is because there is an entry in the /etc/services file mapping these names to their respective numbers. In the above listing, the numbers 32768, 32794, and 893 ought to be investigated because they’re not registered as being “known” entities, which is the sort of thing you must look for. While you’re looking, I will check out dev.localdomain…

Netstat is the “king” of blackhat detectors because some service will probably be set up that wants to do stuff on the Internet, and netstat is the utility that will show you. Blackhats know this, and tend to modify or delete netstat. If they do, the symptom is that netstat will be missing, or if present, will often core dump.

The Queen of Blackhat Detectors

Check your process listing for processes that don’t belong. Telltale signs are processes with oddball pathnames or that are using a lot of CPU or Memory. Use the widest options for your ps listing: ps auxwww | sort | less. Most blackhat programs stand out like sore thumbs. Some, however, might masquerade as normal daemons. For example, Red Hat 7.2 runs xinetd, not inetd, so if you see inetd, you know something’s not right. Another ruse is to use the same name as an existing daemon, but run it from an odd place. For example, crond should be running from /usr/sbin (system programs/daemons), but it’d look suspicious if it were to be run from /usr/bin (user programs).

Worse yet are programs that are run from ./. Unless you know who is running them, and can verify them, its unusual for root to need to run ./ for a system program. Linux has an additional advantage over blackhats with regards to ps. The ps executable file is directly related to the specific version of your kernel. If the kernel changes, then ps changes. Furthermore, if ps changes and the kernel doesn’t, ps will coredump. If ps coredumps, it's likely been hacked.

Other Considerations

Every blackhat loves to squirrel things away. Some are really good and some don’t do it at all. Count yourself lucky if they don’t remove their tracks — this makes it a lot easier to repair the damage. I once found an entire directory of nastiness in /usr/local/hack. It had source code, config files, and best of all, the log files from the daemon that was running. In short, the log files were gold since I was able to easily match up my netstat listings to the log entries and figure out the blackhat’s complete scheme. I attribute this type of break-in to “script kiddies".

Anecdotal Lesson #5: Look for the stupid things first — many of today’s blackhats don’t know much about what they’re doing, but know enough to run a script. You’ll save a bundle of time.

On the flip side, there are also some pretty darned clever places to hide things. My favorite was when someone hid a directory in the /dev directory. Most things in the /dev directory are either block (b) or character (c) devices. The “ls –l” listing looks like this:


brw-rw---- 1 root disk 8, 0 Aug 30 16:30 sda
brw-rw---- 1 root disk 8, 1 Aug 30 16:30 sda1
brw-rw---- 1 root disk 8, 2 Aug 30 16:30 sda2
.
.
crw-rw-rw- 1 root tty 2, 176 Aug 30 16:30 ptya0
crw-rw-rw- 1 root tty 2, 177 Aug 30 16:30 ptya1
crw-rw-rw- 1 root tty 2, 178 Aug 30 16:30 ptya2
This particular intruder decided to create a hacker directory called /dev/sda01. In the colorized version of ls, however, it stuck out like a sore thumb. In black and white displays, it would be easily overlooked. Here’s what that directory looked like (along with a valid /dev/sda1):


brw-rw---- 1 root disk 8, 1 Aug 30 16:30 sda1
drwxrw---- 2 root root 4096 Feb 14 15:12 sda01
As you can see, the listing is totally different, but without the –l option on ls, it's pretty easy to miss:

sda sda01 sda1 sda10 sda11 sda12 sda13 sda14 sda15 sda16 sda17 sda18 sda19 sda2

These are all illustrative examples of the kinds of things to look out for. Don’t assume that your .ssh directory is safe, or that any place is “safe”. Once you have root permissions, every place is as good as another.

Last One There Is a Blackhat!

The command, last , gives you a pretty nice display of who logged onto your system in the past. If you use last -a , you’ll get a complete display of the hostname from where your blackhats arrived. Incidentally, the -a option is only available under Linux — Solaris, MacOS X, and FreeBSD have not implemented this handy formatting option. The last command is just a nice way of extracting information out of the /var/log/wtmp and /var/log/wtmpx files. Since it's such a damning display, the two wtmp files and to a lesser extent, the last command, are usually deleted as the last thing the blackhat does before logging out. Here’s a short listing (last will go on for a long time if you don’t use head):

[root@dev root]# last -a | head
root pts/17 Thu Feb 14 16:04 still logged in jeffk.localdomain
cheryl pts/4 Thu Feb 14 16:03 still logged in riptide.localdomain
sailh pts/13 Thu Feb 14 16:00 still logged in station2.nasa.gov
cheng pts/26 Thu Feb 14 15:52 still logged in
cheng pts/0 Thu Feb 14 15:52 - 15:52 (00:00) auction2
cheryl pts/4 Thu Feb 14 15:52 - 16:01 (00:09) riptide.localdomain
fortune pts/13 Thu Feb 14 15:44 - 15:53 (00:09) eecsdial-gw
andrew pts/4 Thu Feb 14 15:41 - 15:44 (00:03) seacow.localdomain
sowyarn pts/13 Thu Feb 14 15:21 - 15:26 (00:04) current.engin
sharp pts/27 Thu Feb 14 15:13 - 15:31 (00:18 ) skal.localdomain

Your Systems Nanny: syslogd

All UNIX flavors these days have some sort of syslog daemon running. Syslog is a handy daemon in that it provides programmers with a simple way to save something to a system log file, usually with a single function call. As a result, syslog calls are the preferred way to do logging in a UNIX application. Syslog is handy from an administrator’s perspective because it provides a way to separate all of those log entries into different files. Thus, you don’t have to sift through a gigabyte of text file to find the blackhat clues you’re looking for.

Here’s a sampling of /var/log/messages that is output by syslog:

Feb 28 00:05:24 server1 rpc.mountd: refused mount request from sligo. \
localdomain for /k1 (/): noexport entry
Feb 28 00:05:24 server1 rpc.mountd: authenticated mount request from \
123.45.6.78:1023 for /p1 (/p1)
Feb 28 00:08:01 server1 login(pam_unix)[7895]: check pass; user unknown
Feb 28 00:08:01 server1 login(pam_unix)[7895]: authentication failure; \
logname= uid=0 euid=0 tty=pts/7ruser= rhost=dial-148-240-106-8.zone-3.\
dial.net.mx
Feb 28 00:08:03 server1 login[7895]: FAILED LOGIN 1 FROM dial-148-240-106-8.\
zone-3.dial.net.mx FOR guest, Authentication failure
Feb 28 00:08:14 server1 rpc.mountd: export request from 123.45.6.89
Feb 28 00:09:07 server1 imapd[3270]: Logout user=jim host=azure.localdomain \
[123.4.5.6]
There are a couple of interesting things about this listing:

• Line 1: A request to NFS mount a partition is refused — sligo wasn’t authorized, and could be suspicious.

• Lines 3-5: Someone tried to login as a user “guest”. Complete connection details are given. This is likely a blackhat probing for weak security on a target system. You’ll notice that PAM caught the probe, not the login program.

• Line 7: An IMAP connection is finished and closed off. It's handy to know who is using the system and when, so that if something suspicious is done at a time you know isn’t normal, you have a lead to follow. These sorts of entries help to give you a sense of what is and isn’t normal on your system.

The text files produced by syslog are found in one place on a system, but that place varies from OS to OS. Common places to look for files is in /var/log and /var/adm, but could be anywhere. In Linux, /etc/syslog.conf is the configuration file and will tell you where to look. Here’s a snippet of a syslog.conf file:

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
*.info;mail.none;authpriv.none;cron.none @slserve.localdomain

# The authpriv file has restricted access.
authpriv.* /var/log/secure
authpriv.* @slserve.localdomain
# Log all the mail messages in one place.
mail. * /var/log/maillog
The interesting things in this file:

• All of the info- level messages (*.info) and the none- level messages are logged to /var/log/messages. These are “common” messages that are often not very important.

• All authpriv-class messages are logged to /var/log/secure. This file will contain all of your authentication messages, both normal and abnormal.

• All mail-class messages are logged to /var/log/maillog. Mail is very chatty, so it's a good idea to section these off. Note, however, that the Sendmail daemon is a common break-in point, so do watch these logs for improprieties.

• And finally, note the duplicated lines with @slserver.localdomain destinations. What the heck are those? I'll explain.

The last and coolest thing that syslog can do is to send a copy to a syslog server machine. Why would we want to do that, since log files are very system-specific? If you configure your syslog server machine’s firewalling to be super-secure, and only let in ports 514 (the syslog port) and 22 (ssh), then you have a copy of all of the system logs from your original server that are guaranteed to be free of tampering from blackhats. It a nice, redundant, defense technique that’ll save you a lot of anguish when the blackhat’s scripts have blasted your /var/log directory into oblivion.

Blackhat Prevention

A word about your Linux install…

I’m a firm believer in the use of a package manager, specifically, the Red Hat Package Manager (RPM). Use of a package manager means that my installs will be “stock” and not very customized, but they have one huge advantage with respect to forensic investigation — they present a known and verifiable installation of the operating system and its tools. This assumes that your RPM database, found in /var/lib/rpm, is intact, so keeping “good” copies of it isn’t a bad idea at all. Tarballs (.tar.gz or .tgz) are not sufficient, because they provide no consistent, systematic way of verifying the integrity of the files once they’re installed. Unless you’re willing to create a database of MD5 checksums of every file you install, keep tar at arm’s length. The Solaris, Debian Linux, and BSD variants all have package tool systems, but as of this writing, they do not have verification features. Once you install, keep in mind (or in a file somewhere) which files you had to customize and non-RPM programs you had to install after the fact (which should be installed in /usr/local). This will come in handy once you start your pursuit of your next blackhat.

Remember: You need to know what your system looks like under normal conditions.

Swiss-cheese packages to avoid

Most packages available for Linux are reasonably secure, but there are some that are known to be full of security holes. Here’s an annotated list of tools and services to avoid if at all possible:

NFS (nfs-utils, am-utils). This is UNIX network file- sharing mechanism. Its UDP-based and, therefore, not very reliable. Am-utils are a series of programs that go with NFS for automounting support. Both are porous and if not needed, shouldn’t be installed.

YellowPages (ypbind, ypserv, yp-tools). These provide shared configuration services. They’re handy, but full of security holes.

Non-ssh remote tools (rsh, rsh-server, rdate, rdist, rwho, rwall, rcp). Most of these have ssh-capable equivalents, so you shouldn’t need them. Ssh and scp provide 99% of the functionality of these insecure packages.

Anonymous FTP. Any FTP daemon can do anonymous access, but avoid installing config files unless you have a really good reason to open up your system.

Iron-clad packages to seek out

Most installs do pretty well at installing “the good stuff”, but nonetheless, here are some packages that you should seek out.

OpenSSH (openssh, openssh-askpass, openssh-clients, openssh-server, openssl). These utilities are the gold-standard of security. Know them, use them, love them. They offer full encryption as well as handy features like compression for scp file transfers, thereby saving you network bandwidth, too.

Firewalling (ipchains, iptables or ipfw). Depending on your kernel and OS, one or more of these firewalling packages will be available.

TCPwrappers (tcp_wrappers). These config files allow you to control remote access to any tcp-based service even more precisely than the firewall can. It's essentially a “second line of defense” that’s easy to erect and maintain.

Simple Is Secure

Keep in mind that you need to strike a balance between usability and security. You will need to be able to change the configuration settings easily for maintenance, but must do so without introducing any new security holes. Security software that is simple to use is usually complex underneath and can often introduce additional, subtle, security holes in the process of fixing obvious ones. If a security system installs a daemon and makes you open a tcp or udp port to help you close off holes in telnet or ftp, you really haven’t gained much. The daemon needs securing, and is now a potential exploit. The additional ports and the program(s) that service them are two more holes the blackhat has to work with. Ideal solutions use as few pieces as possible so that you can understand and maintain the security model.

Linux Firewalling: ipchains and/or iptables

Many Linux distributions include some type of default firewalling. Red Hat 7.2 offers firewall settings of High, Medium, or nothing as part of its install. I recommend choosing the Medium (default) setting, because it’s a reasonable starting place. Do not assume this is the last time you’ll need to look at the firewall settings! Red Hat should be commended as having taken an important step, but it's really about all they can do without knowing the exact makeup of your computing environment.

“Firewalling” in Linux refers to the process of deciding which Ethernet packets are accepted or denied in your system for both incoming and outgoing packets. “Packet Filtering” or “Ethernet Filtering” are common names for the same thing. In Linux, the firewalling occurs at the level of the Ethernet device driver, which is the only place it can really occur and be safe from circumvention.

There are two commonly used firewalling methods and how each is used depends on the kernel version you have. If you don’t know what version of the kernel you’re using, uname -r will list out either a 2.2.x number or a 2.4.x number. Ipchains is the firewalling method used in 2.2 and 2.4 kernels. Iptables is used only in the 2.4 kernels. Iptables is purportedly the more flexible and powerful of the two methods, but as a result, requires more attention to detail when configuring it. Ipchains is pretty easy, but is run in an emulation mode in 2.4 kernels, and as a result, is not 100% functional (but functional enough for most firewalling situations). I will describe ipchains only, because iptables is so close. The ipchains configuration file is found in /etc/sysconfig/ipchains on Red Hat systems. A sample file is below with an explanation following.

1 # Firewall configuration written by lokki
2 # Manual customization of this file is not recommended.
3 # Note: ifup-post will punch the current nameservers through the
4 # firewall; such entries will *not* be listed here.
5 :input ACCEPT
6 :forward ACCEPT
7 :output ACCEPT
8
9 # BANNED DOMAINS
10 -A input -p tcp -s 12.41.160.233/255.255.0.0 -d 0/0 0:65535 -y -j DENY
11 -A input -p tcp -s 24.183.171.108/255.255.0.0 -d 0/0 0:65535 -y -j DENY
12 -A input -p tcp -s 63.116.175.200/255.255.0.0 -d 0/0 0:65535 -y -j DENY
13
14 # standard services
15 -A input -s 0/0 -d 0/0 21 -p tcp -y -j ACCEPT
16 -A input -s 0/0 -d 0/0 22 -p tcp -y -j ACCEPT
17 -A input -s 0/0 -d 0/0 23 -p tcp -y -j ACCEPT
18 -A input -s 0/0 -d 0/0 25 -p tcp -y -j ACCEPT
19 -A input -s 0/0 -d 0/0 80 -p tcp -y -j ACCEPT
20
21 # DNS
22 -A input -s 123.45.67.22 53 -d 0/0 -p udp -j ACCEPT
23
24 # AFP: Apple File sharing over TCP/IP: 548
25 -A input -s 0/0 -d 0/0 548 -p tcp -y -j ACCEPT
26
27 # Samba I/O: 139
28 # caution: currently open to all, but should be protected by EECS routers
29 -A input -s 0/0 -d 0/0 139 -p tcp -y -j ACCEPT
30 -A input -s 0/0 -d 0/0 139 -p udp -j ACCEPT
31
32 # POP
33 -A input -s 0/0 -d 0/0 110 -p tcp -y -j ACCEPT
34
35 # IMAP
36 -A input -s 0/0 -d 0/0 143 -p tcp -y -j ACCEPT
37 -A input -s 0/0 -d 0/0 992:995 -p tcp -y -j ACCEPT
38
39 # needed for NFS mounting
40 -A input -s 123.45.67.208 -d 123.45.67.202 -j ACCEPT
41
42 # Printer syslogging from
43 -A input -p tcp -s 123.45.67.52 -d 123.45.67.202 514 -y -j ACCEPT
44
45 # CVS
46 -A input -p tcp -s 123.45.67.255/255.255.255.0 -d 0/0 2401 -y -j ACCEPT
47
48 -A input -s 0/0 -d 0/0 -i lo -j ACCEPT
49 #-A input -p tcp -s 0/0 -d 0/0 0:1023 -y -j DENY
50 ##-A input -p tcp -s 0/0 -d 0/0 2049 -y -j DENY
51 #-A input -p udp -s 0/0 -d 0/0 0:1023 -j DENY
52 ##-A input -p udp -s 0/0 -d 0/0 2049 -j DENY
53 #-A input -p tcp -s 0/0 -d 0/0 6000:6009 -y -j DENY
54 #-A input -p tcp -s 0/0 -d 0/0 6667 -y -j DENY
55 #-A input -p udp -s 0/0 -d 0/0 6667 -j DENY
56 #-A input -p tcp -s 0/0 -d 0/0 7100 -y -j DENY
57
58 # This line may be needed for ";fragmented packets";. I don't quite know
59 # whether it is really needed or not, but I have read some stuff that
60 # it is required for NFS
61 -A input -f -j ACCEPT
62
63 # throw everything else out
64 -A input -s 0/0 -d 0/0 -p tcp -y -j DENY
65 -A input -s 0/0 -d 0/0 -p udp -j DENY
The ipchains configuration file is nothing more than a series of commands to give to the ipchains command, hence the rather strange look of the configuration file. Here’s a closer look at /etc/sysconfig/ipchains:

• At the end of each command line (everything except lines 5-7), you can tell ipchains to do one of three things: ACCEPT, DENY, or REJECT. I prefer not to use REJECTs, because they will return a message informing the blackhat that I’ve blocked him, while a DENY will simply drop the packet without providing any additional information.

• The –s and –d options let you specify the source and destination for a packet, respectively. There are two ways to specify it: either with just an IP address, or if you want to use a mask, you can specify whole domains. Lines 10-12 mask off entire domains as the source of a packet, such that the last two numbers of the packet’s IP address are ignored.

• Lines 5-7: Firewalls handing input, output, and forwarding of Ethernet packets. Forwarding is used to configure your Linux machine as a router. This is often part of a larger technique known as “IP Masquerading”. Note that in 2.4 kernels, only input and output commands work. Leave lines 5-7 as is, because you can configure everything else without use of them. For our purposes, we’re using the “:output” section.

• Lines 9-12: I configure my firewalls such that if an IP address or domain is known to be bad, I don’t even let them get into the decision process.

• Lines 15-17 let standard services get through. I let telnet, ssh, ftp, Sendmail, and http through without question. telnet and ftp are obviously questionable, but on my particular machines, they’re virtually required because of users that come from outside our network.

• Line 22 allows DNS request info to return back to my server.

• Line 25 allows Apple File Sharing to work on my server.

• Lines 29 and 30 let Windows File Sharing through.

• Lines 32-38 allow mail connections via IMAP and POP through.

• Lines 40 and 61 allow NFS File Sharing to function properly.

• Line 43: This particular machine receives syslog information from our printers, as it is a printer server, so I have to permit port 514 for those printers. Note that if you set up a syslog server as mentioned above, you’ll need to open up port 514 for each of those machines. Syslog is not a terribly secure protocol, so it's very important to permit access on a machine-by-machine basis.

• Line 46: My server is a CVS server that utilizes the pserver protocol, so port 2401 must be opened. Note that I opened it only for my subnet. If you have remote users outside of a subnet, you might want to permit them either individually, or by domain.

• Lines 64 and 65 handle any packet that hasn’t already been allowed or denied. In this case, I DENY any packets that don’t meet my ACCEPT criteria, thus making it a lot harder for a blackhat to come in on oddball ports.

Once you set everything up the way you want it, you can update the firewalling to conform to your config file by restarting ipchains:

/etc/rc.d/init.d/ipchains restart

PAM (Pluggable Authentication Modules)

I’m not going to pretend to be any kind of expert on PAM, because frankly, it's the one system structure that I constantly grumble at. However, I can’t trash it, because it catches so many nasty things. In a nutshell, PAM is really sweet when it keeps blackhats out, but when configuring it, PAM can be a nightmare. All PAM config files exist in /etc/pam.d in Red Hat distributions. Here’s the /etc/pam.d/login file, which is used whenever you interactively log into the system:

#%PAM-1.0
auth required /lib/security/pam_securetty.so
auth required /lib/security/pam_stack.so service=system-auth
auth required /lib/security/pam_nologin.so
account required /lib/security/pam_stack.so service=system-auth
password required /lib/security/pam_stack.so service=system-auth
session required /lib/security/pam_stack.so service=system-auth
session optional /lib/security/pam_console.so
Generally, to solve most PAM issues, you simply need to fiddle with the middle column settings so that something might be optional rather than required in order to let your situation get through. I find the arrangement of these things pretty confusing, so I tend to leave them alone. Use “man pam ” for reference. The man page is very good at explaining what you see in the config files, but even the man pages note that “it is not of primary importance to understand the internal behavior of the Linux-PAM library.” The default settings work pretty well in 99% of the cases under Red Hat 7.2, and I breathe easier knowing I haven’t had to mess with them.

Sources for More Information

1. CERT Coordination Center (http://www.cert.org): These are the folks who track nasties and send out advisories. Get on their mailing list(s) for “weather reports” of hack storms.

2. Red Hat, Inc. (http://www.redhat.com): Excellent source of info on tweaking Red Hat Linux. Look here for security alerts and replacement RPMs for remedying the problems.

3. Debian (http://www.debian.org): If you’re not using Red Hat, then you’re probably on a GNU Debian or derivative. Look here for info, security alerts, fixes, etc.

4. BrickHouse by Brian Hill (http://personalpages.tds.net/~brian_hill): This is a $25 shareware GUI front-end for MacOS 10.x ipfw command that is worth its weight in gold. If you run MacOS 10.x, get it and pay for it!

5. Maximum Linux Security by Anonymous, published by Sams. ISBN 0-672-31670-6. Nearly 750 pages of Linux security essentials including a CD-ROM. It’s a no-nonsense book that acts as both a tutorial and a reference. http://www.samspublishing.com or your favorite bookstores (I use BookPool for all of my technical books: (http://www.bookpool.com) because they regularly have everything technical under one roof and at competitive prices).

6. UNIX System Administration Handbook by Evi Nemeth, Garth Snyder, Trent Hein, et al., Third Edition. ISBN #0130206016. An essential all-in-one UNIX systems admin guide. Covers all major UNIXes well in a reasonably compact size.

7. OpenSSH. (http://www.openssh.org): OpenSSH also has libraries for SSL and other security-related programs.

8. Network Simplicity (http://www.networksimplicity.com/openssh): This has an excellent build for Windows users that is easy to install, yet standard.

9. OpenBSD (http://www.openbsd.org): If Linux security leaves you flat, check out OpenBSD — it's secure, out-of-the-box, and very stable. It has a Linux-compatibility mode, so most, if not all, Linux software can function on OpenBSD. Be forewarned, however, that this distribution is not as polished as most Linux distributions, so it's not for the uninitiated.

10. WinSCP (http://winscp.vse.cz/): If you're a Windows user, you'll want a decent client to use SCP instead of FTPing files to your secure system. It has a familiar twin-paned interface that most Windows users will feel at home with.

11. MacSFTP (http://www.macssh.com): This is a Carbonized SCP program, despite the title. This is about as close to Fetch as you can get, but with SCP/SFTP capability. It's shareware, but the price is only $25 — well worth it.

12. Apache (http://www.apache.org): Apache is king when it comes to solid, secure Web servers for all OSes. I recently replaced IIS with an Apache server and it was both faster and bulletproof — my users rejoiced.