12 tips for securing your linux systems

From time to time I get amazed how people can setup their production servers. At the smallish development companies there is no real system administrator available to setup the systems and to keep them up to date. Now I’ve seen systems that have been setup ranging from “somebody with sufficient knowledge” to “this-was-setup-by-the-janitor” and everything in between. So, if you are a “programmer who knows a bit about Linux because you’re using Ubuntu”, but you have no real idea on how to SECURELY setup a system, here are some tips.

Word of caution

These tips are NOT in any order, nor does it mean that after following these tips it means your boxes are secure. They are just simple reminders about things I see in practice that are done wrong. The best advice I can give to EVERYBODY who deals with production systems is to hire (be it on a freelance basis) an expert in system administration and security. It would be devastated for you and/or your companies reputation if you create a wonderful highly secure web application and all information gets compromised nevertheless because your MySQL database was wide open on your production servers. These are not area’s you can take for granted or let some junior developer handle… really…

Tip 1: Use the correct distribution

I’ve seen desktop-versions of popular distributions being installed on production servers. I’ve seen X-windows / KDE installed. Make sure you use a “server” distribution. But not even all server distributions are all equal. Some distributions are very stable, but use old versions of the most common software. Other distributions will have all the latest releases, but this can result in a unstable system.

I prefect Debian for server systems. A basic install is very small and only the most necessary packages are installed. When you need more, you need to add them yourself which is what you want on a server system. However, Debian has the disadvantage of being “too stable”, which means that it takes a long time for new version of packages will come out. There are many 3rd party repositories out there with new releases, but you have to be careful not to sacrifice the stability of your server.

Tip 2: Minimize access to the system

An unreachable system is an unhackable system. Since pulling out all the network cables of a production server is not really an option in most cases, the next best thing is to minimize the connections that can be made. Most distributions will setup all kind of services to the outside world: smtp, pop3, dns, ident, http and many many others.

debian-jth:~$ sudo netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:59814           0.0.0.0:*               LISTEN      2615/perl       
tcp        0      0 0.0.0.0:38856           0.0.0.0:*               LISTEN      1845/rpc.statd  
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      2154/mysqld     
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      2618/memcached  
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1834/portmap    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2077/sshd       
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      2261/postgres   
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2605/exim4      
tcp        0      0 0.0.0.0:7003            0.0.0.0:*               LISTEN      2615/perl       
tcp6       0      0 :::139                  :::*                    LISTEN      2636/smbd       
tcp6       0      0 :::80                   :::*                    LISTEN      2717/apache2    
tcp6       0      0 :::22                   :::*                    LISTEN      2077/sshd       
tcp6       0      0 ::1:5432                :::*                    LISTEN      2261/postgres   
tcp6       0      0 :::445                  :::*                    LISTEN      2636/smbd       
udp        0      0 192.168.1.102:137       0.0.0.0:*                           2634/nmbd       
udp        0      0 192.168.56.101:137      0.0.0.0:*                           2634/nmbd       
udp        0      0 0.0.0.0:137             0.0.0.0:*                           2634/nmbd       
udp        0      0 192.168.1.102:138       0.0.0.0:*                           2634/nmbd       
udp        0      0 192.168.56.101:138      0.0.0.0:*                           2634/nmbd       
udp        0      0 0.0.0.0:138             0.0.0.0:*                           2634/nmbd       
udp        0      0 0.0.0.0:38676           0.0.0.0:*                           2065/avahi-daemon:
udp        0      0 0.0.0.0:53397           0.0.0.0:*                           1845/rpc.statd  
udp        0      0 0.0.0.0:68              0.0.0.0:*                           1793/dhclient3  
udp        0      0 0.0.0.0:68              0.0.0.0:*                           1730/dhclient3  
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           2065/avahi-daemon:
udp        0      0 0.0.0.0:749             0.0.0.0:*                           1845/rpc.statd  
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1834/portmap    
udp        0      0 192.168.56.101:123      0.0.0.0:*                           2650/ntpd       
udp        0      0 192.168.1.102:123       0.0.0.0:*                           2650/ntpd       
udp        0      0 127.0.0.1:123           0.0.0.0:*                           2650/ntpd       
udp        0      0 0.0.0.0:123             0.0.0.0:*                           2650/ntpd       
udp6       0      0 :::59421                :::*                                2065/avahi-daemon:
udp6       0      0 :::5353                 :::*                                2065/avahi-daemon:
udp6       0      0 ::1:123                 :::*                                2650/ntpd       
udp6       0      0 fe80::a00:27ff:fe62:123 :::*                                2650/ntpd       
udp6       0      0 fe80::a00:27ff:fe7e:123 :::*                                2650/ntpd       
udp6       0      0 :::123                  :::*                                2650/ntpd       
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     5406     2053/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     5386     2043/acpid          /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     6143     2261/postgres       /var/run/postgresql/.s.PGSQL.5432
unix  2      [ ACC ]     STREAM     LISTENING     5609     2154/mysqld         /var/run/mysqld/mysqld.sock
unix  2      [ ACC ]     STREAM     LISTENING     5440     2065/avahi-daemon:  /var/run/avahi-daemon/socket

Services listening to 0.0.0.0 means they will listen on ALL interfaces. If you need services to be used internally, like a MySQL server, make sure you bind that service to a specific interface AND/OR (preferably AND) make sure your firewall blocks incoming connections from the outside interfaces as well. In this example you see that there is a perl-process listening on all interfaces on port  59814. In this case its a gearman server so we shouldn’t be able to connect to this process from the outside so there is room for improvement if this was a production system.

Make sure your system does not listen to more services it needs to, and use tools like nmap to scan your system to make sure only the ports open that are allowed to be open.

Tip 3: Do not use “shared” accounts

Never have two or more people share the same user account. It’s very simple to give everybody their own account but again, sometimes it is a necessity. For example when you do not own the production server and you are a “guest” on it. You probably only get 1 useraccount that you have to share with the rest of the company.

The reason why you shouldn’t use shared accounts are:

• If person A or person B losses the password, the account is compromised. Multiply this by the fact that every employee has access to that account and password.
• If person A gets fired or quits its job, the password must be changed. This means that once an employee leaves the company, everybody much change the password.
• There is no way of telling who entered the production server. It could be person A, or B, or somebody else..
• Everybody gets all permissions. You cannot make any distinctions between users.

If you can’t avoid shared accounts, make sure you can only login through public key authentication. You are able to login to the same account, but everybody gets their own “password”. This means that disabling one user doesn’t mean everybody else needs a new password and secondly, it is logged WHO has logged in (or at least, with what key).

Tip 4: Log everything to a remote server

Make sure log-files about the status of your server are send directly to a remote log-server. IF a server gets compromised, at least you have the log-files to find some information about what has happened. If you keep the log-files on the same server, it’s easy enough for a hacker to remove all his traces so you won’t know anything on what has happened. Everything that is capable of syslogging (which everything should be.. it’s there for a reason) can be filtered, duplicated and send to multiple other systems. Use it instead of logging everything to your own logfiles.

If something has happened, check out the logs from the remote logserver and see what has happened. If anything at least you can prevent somebody else hacking your system the same way (everybody can have their boxes hacked, it happens… if it happens twice the same way, then it’s plain stupid)..

Tip 5: SUDO instead of root

Now I’m personally do not agree with this, but for most people it actually is good advice, so I add it to my list anyway. Instead of logging in as root directly, issue your commands through sudo. First of all, you can decide which users can issue which commands. For instance, some users you wouldn’t trust with complete root-access, but they are capable of editing and restarting the web-server. You can use /etc/sudoers en visudo to achieve this.

Tip 6: Whitelist your firewall

Some distributions have a very ridiculous default firewall policy. The default policy of those systems is to accept everything. Some will argue that this doesn’t matter since the last line of those firewall rules would be “drop everything didn’t match anything else before me”, but this is exactly what policies are for in the first place. Furthermore, firewall rules are normally added and the end of the line, so those newly added lines will never match.

So instead of having a default ACCEPT policy, make sure you DROP everything you didn’t explicitly match.

It has a few advantages:

• If you “forget” a blacklist rule, you don’t destroy security but merely adding “too much”.
• If you accidentally clean out your iptables list, at least everything gets blocked. This includes you too, so you must take additional action like having additional way of accessing your system like a serial connection or KVMoverIP switch.
• It’s the correct way of treating a firewall: you should define who get ACCESS, not who doesn’t.. the latter list is always much larger.

Tip 7: Don’t add fixed ip’s to your firewall

It looks like a good idea: we deny access to SSH unless you come from a certain range. And on the surface it IS a good idea. Even though SSH is safe, you don’t want to have thousands of attempts by unknown users to try to access your system? What if one of those attempts succeed? And what if that password is used throughout many other systems? By the time you figure out somethings wrong, you are already too late.

So instead we setup the firewall so only certain IP’s are allowed. We can include the static IP from our company and some IP’s from employee’s at home so they can login in case something happens outside business-hours. Oh, and we need to add a complete range or two since some of us don’t have static IP’s.. and maybe some IP’s for some customers who on occasion need to use FTP, and of course, they are not static as well….  As you can image, this can become a mess quite quickly so we don’t want to do it with a firewall rules.

Another option could be 3rd party tools like denyhosts, fail2ban etc..  I would advise against those systems because they use up too much memory. I’ve seen memory usages of 100M+ of denyhost applications. We could use those megabytes for a much better cause (memcache, mysql etc).  Furthermore, they are working from userland, which means the connection is made and the harm is already done. And the most “annoying” reason to not use these tools is the many “false positives” you can get. I occasionally try to login with a false password: either my caps-lock is on (I don’t spend that much time looking at my keyboard), or I’m typing a wrong password. So after 3 attempts or so not only I, but the whole IP (which normally means: the complete company) gets banned from the server. Great.. now I have to login into another system, see if we can login from there and remove the ban manually..

So what is a GOOD solution then?

It’s called port knocking. Basically what it means is that we have to “knock” on a port (say, port 15000) first, after which the initial ssh-port (port 22) will open itself for that IP-adres for a certain amount of time in order to create a connection. Port-knocking is very easy to implement with the help of iptables and has the following advantages:

• No need for bloated 3rd party userland tools.
• It will whitelist IP’s instead of blacklisting IP’s.
• It doesn’t matter from WHERE you want to connect, so no big table full with IP’s. So you can use it literally everywhere without compromising its goal.
• You can create very easy to very complex knock-sequences with multiple ports if you like, but a simple knock is enough to minimize the unauthorized access-attempts with almost 100%

Setting up port knocking:

iptables -A INPUT -i eth0 -p tcp -m tcp --dport 16000 -m recent --set --name SSH1 --rsource
iptables -A INPUT -i eth0 -p tcp -m tcp --dport 22 --tcp-flags SYN,RST,ACK SYN -m recent --rcheck --seconds 120 --name SSH1 --rsource -j ACCEPT
iptables -A INPUT -i eth0 -p tcp -m tcp --dport 22 --tcp-flags SYN,RST,ACK SYN -j DROP
iptables-A INPUT -i eth0 -p tcp -m tcp --dport 16000 -j REJECT

Line 1: create a “recent” entry labeled “SSH1” for the remote source IP when a connect is made to port 16000.

Line 2: when somebody creates a initial TCP connection to port 22 (ssh), AND the remote source has match in the “recent” table labeled SSH1 in the past 120 secondes, then this connection is ALLOWED. The “SYN,RST,ACK SYN” tcp-flags means that the SYN-flag must be set, but the ACK and RST flags must NOT be set (we don’t care about other flags in this case).

Line 3: Otherwise, all other connections are DROPPED

Line 4: We REJECT connections to port 16000, which will tell telnet connections that this port is unavailable. It will prevent telnet-sessions to “hang” since there is nothing on port 16000 to listen at.

(this example does not use connection tracking. It might be even better to implement connection-tracking in this as well).

To see some information about the current “port-knock” state, take a look a the /proc/net/ipt_recent directory.

Tip 8: Use iptables firewall modules

In the last examples, we have used the “recent” iptables module. But there are many others which can be used to secure your system. The most important one is “connection tracking” (conntrack). As soon as you decide a certain connection is allowed inside, all other packets related to that connection is automatically allowed. For normal tcp-connections like SSH this is not really an advantage, but it will for multi-port protocols like FTP for instance. Instead of opening up all your high ports (I’ve seen firewalls that open up port 1024 and higher just to make FTP work), we add it to the connection tracker. It will figure out by itself what port must be opened (and only for that particular IP) when an FTP connection is made. It will avoid opening up your firewall to wide.

There are many others like “time”, where the firewall will only allow packets between a certain time period. “Connrate” and “limit”, where your firewall will throttle the amount of connections made and it’s even possible to add a “geoip” module, which makes your firewall geo-aware so you can block out all connections coming from a certain country. The possibilities are endless…

Tip 9: Minimize “plaintext” protocols

Use SFTP instead of FTP. Use IMAPS instead of IMAP etc. Most plain-text protocols have a “secure” counterpart nowadays and unless you really can’t otherwise, I suggest you use the secure version.

Tip 10: Monitor your system

Systems should be monitored, but I occasionally see monitoring packages that take up half the system resources. There are plenty lightweight systems out there that do a marvelous job.

• Munin is a simple yet very powerful monitoring system. You add a “munin-node” to every server you like to monitor which transfers all data back to a global monitoring server so you have a complete overview of all your servers at once.
• Monit will warn you when something happens to processes on your system. There are great howto’s on installing munin+monit on your systems at howtoforge.
• Logwatch/logcheck are simple tools that will scan your logs for certain keywords and reports them.
• Tripwire is an intrusion detection system (IDS) that will detect, well, intrusions into your system.
• Monyog is a powerful (commercial) package that can monitor your MySQL databases. There is no need to install client-side software. A MySQL connection or SSH account is enough to get it working.

Tip 11: Keep up-to-date

Once your server is up and running correctly, your task is not finished. From time to time you must update your software to patch security leaks. The longer you wait, the more chance your system will be compromised. Make sure you spend time in updating your systems periodically, and make sure high-priority patches gets installed as soon as possible. Don’t wait a week just because you have deadlines to meet. With a compromised server, those deadlines are the least of your worries.

Tip 12: Know your system

Nothing compromises security more than not knowing your systems. It’s true: how can you sure something if you don’t know how it works in the first place? So if you don’t know (enough) about Linux systems, go and learn more about it. Using some sane reasoning and knowledge is probably the best defense against security breaches.

Conclusion

I get into discussions with (php) developers about how system administration is not “our thing” so we should not worry about them. I don’t really get that. First of all, it is my thing because I’m both a developer AND an administrator, but more important: It REALLY us our thing. Our customers do NOT pay us for developing an web-application, but they pay us to create something they can use from the web and this also includes getting it online and making sure it keeps on working and is secure. If they only want you to deliver a tarball with source code than fine, you don’t need any system administration but I doubt that this is the case at your company.

You can outsource administration to a external party but that comes with a very high cost. Especially when you have a lot of servers (even small companies can easily have 50+ servers) so it actually might be more cost-efficient to hire a small staff of administrators.

But whatever you do,.. never underestimate the necessity of administrators. They are just as important as having a QA-team, creating unit-tests, flossing regularly, finessing, eating healthy and anything else that you know is good for you, but which you don’t do anyway..