What do Linux Admins feel is the Minimum needs to know?

[joseph]

[Replied by Steve Holdoway] That for the 99 times you get away with cutting a corner, cleaning up after the 100th costs far, far more. It's not so much about what you know - although as Richard mentioned, you need to be able to manage a mail server - but it's about attitude.

Everything *will* go wrong, and you need to be prepared for it. Your servers *will* get hacked, how will you protect, recognise, and act as soon as poss?

On top of that, it is imperative that you know what's happening on your servers, so monitoring is extremely important. I use munin ( Munin - Trac ) to keep me informed an to plan ahead.

Finally, a good sysadm will plan upgrades so that they are as invisible as possible to the user base - less retraining, less upset.

Of course, once your systems are running perfectly, and there are no interruptions to service, you will have become invisible, not deemed an important resource, and made redundant! Are you sure you want to make a career out of being a sysadmin???

[joseph]

[replied by Nicholas Accad]
1. Ditch the GUI and live for a month on console alone. If you can do that, you've won half the battle.

2. Make sure you learn to use basic VI, not VIM, not nano, not pico, something that is sure to exist in a rescue image.

3. Learn to read and analyze logs.

4. Crash you machine, overwrite the MBR and see if you can revive it using rescue media, also, learn to use rescue media, always carry multiple, on different media (CD, USB and yes, Floppy), also carry a usb CDROM and Floppy drive.

5. Essential software: Samba, Sendmail/Postfix/Exim (in that order), CUPS (printing), Apache, MySQL and maybe PostgreSQL., Bacula and Amanda for backups.

6. Learn how to use multiple distributions, and go for the basic ones, RHEL and Debian, the others try to make things too easy, not gonna happen to you at 3am in a datacenter in the middle of nowhere.

7. Google, IRC and good-old-fashioned NNTP (newsgroups) are your best friends.

8. Depending on the size of your environment, learn how to configure software iSCSI and FC for storage, learn how to use Nagios (the most common one you'll see, although there are better ones).

9. DO NOT EVER install untrusted software, if your developers are asking for that 100K PHP library that makes their life easier but is made by some guy in Buttf**k, Nowhere, say NO, and insist they use the standard library that the rest of the world uses.

10. Your production machine should have only the bare minimum required for the application to function, do not leave a compiler around, do not leave wget/curl/lynx...etc if you are not using it, do not leave an ftp client, do not leave an SSH client if you do not absolutely need it.

11. Have to system run an integrity check every night, and make sure you have the original MD5 stamps saved offline so you can compare if - no, WHEN the machine gets hacked.

12. Don't trust your users. don't trust ANYONE, root passwords are used once to create an admin account that can sudo, and then they are put in a sealed envelope and kept with a person that has no idea how to use it. Put it in a bank vault where only the CEO can retrieve it.

13. When sending your backup tapes offsite, they will get lost, learn how to encrypt those, learn how to encrypt that DB partition that has the credit card number on it.

14. Everything is your fault. If everything is working fine, you'll be ignored, if you save the day the will say it's your job.

15. Oh.. have fun.

This is essentially what I gleaned after 15 years of Unix system administration with Solaris, Linux, *BSD and some Windows, now I have a nice, relaxing job as a phone monkey

[joseph]

[Dale Gulledge] There's an enormous amount of good advice here in just a few comments. I've been a Linux system administrator. Some of the security measures apply much more strongly if your system is Internet-facing. If you are setting up an internal file server, you may keep some additional tools on it for convenience. If you are running a web app on the wilds of the Internet, clamp it down.

I was administering a web app using JBoss and MySQL. Know enough about those to be comfortable installing, upgrading, starting, stopping and checking up on them.

Nicholas is right about Nagios. You may end up in an environment using something else to monitor your system. If not, knowing Nagios and installing it can make you look like a hero. It'll be the difference between finding out about a problem when someone else notices it and calls you, or finding out minutes or hours earlier when it first occurs.

Know your way around Bash. You may prefer to write various scripts in other languages. That's not a problem. I use Perl a lot. It is worth knowing. I've seen system monitoring tools for apps written in Java and run through Ant. In the end, there's going to be a time when you are going to have to get those running again from the command line.

Make backups. You don't have a real backup until you have tested 3 things:

1) You can restore a single file from it.
2) You can restore all of the data from it.
3) You can restore the entire system.

You can, and possibly might, use separate mechanisms for restoring the OS and 3rd party tools from restoring the stuff you manage locally. That's fine. But the day your hard drive dies and you need that server running again immediately, have a plan for restoring more than just lost. Remember that you want to back up configuration in /etc and probably various stuff in /var as well as /home.

[joseph]

[Eric St-Jean] That's a most excellent list.
It really comes down to practice: boot up an AMI on EC2, make it work as an SSL postfix SMTP server, mail server for local and virtual users, IMAPS server, setup NX and VNC on it so you can remote desktop, set up multiple Apache domains, some in SSL, setup a mod_python app on it, a WSGI django app using MySQL, a RoR app. Set up Lighttpd or nginx load balancing, set up a reverse caching squid proxy. Set up NIS with another host. Set up NFS then benchmark it and research the hell out to make it 10x faster. Set up LVM volumes spanning multiple PV's into a few LVs, at least one of which is then encrypted, and at least one of which is automatically decrypted on a user's login. Then extend that volume by adding a new PV on a different drive - figure out how to turn that into extra space into the encrypted home volume.
Along the way, use vi and vim to edit your files, use bash scripting to automate things. Set up init.d scripts to make things go automatically. Crash your system and restart it. Write some basic python or perl scripts to do things that are too much of a hassle in bash.
Setup nagios or monit monitoring of every single service on a remote host.
Setup Amanda backups using ssh, to S3.

The thing is, most people become amazing sysadmins because they are confronted with problems every single day, and for 15 years they overcome them. It's really hard to tell you how to get there in 3 months, but if you get through Nicholas' list and mine, and get to a point where you know all those things well, you'll be better than a lot of sysadmins out there. And i'm sure there's tons of stuff i'm forgetting. Another list you can look at if you can find it is the general contents of the RHEL certification exam, which makes you set lots of stuff up and troubleshoot things.
If you can do all that in RHEL (you can use CentOS instead) and Debian, you should be fine. Don't use gentoo or slackware - they're neat but you won't find them in a big datacenter. Ubuntu is totally awesome, but basically adds more easy to use layers on top of Debian - just learn Debian and you'll be a great Ubuntu admin as well.
Also: CONTRIBUTE. Look at Debian bug tracking system and http://bugs.centos.org and solve a few of those. You'll get your name out there - i always google people i'm considering, and if you've solved bugs your name will come up - and you'll learn TONS of stuff. Your problem is that you probably don't have many problems to solve so you're looking for things to learn. Solving those bugs will make YOU learn, *and* will also do good for the community. After all, you wouldn't be trying to be a great linux sysadmin if thousands before you hadn't put in the effort to make Linux what it is.

[joseph]

[James Schweitzer] This is a popular topic. I think you have an advantage coming at it from the Windows side. Too many converted Unix administrators (not all) have trouble with the "good enough" principle and the variety of hardware you find in the Linux world. Trying to make Linux be AIX, Solaris or HP-UX isn't the best way to go, in fact the most common underlying hardware (x86) makes Linux sysadmin work closer to Windows than traditional Unix. But be prepared to see Linux running on stuff you don't expect (I see a lot of Power and Mainframes).

While RHEL and SLES dominate in the corporate environment, Ubuntu is a fast riser. They improve at an incredible pace, and while you don't have to change at that pace, you have to be ready to move faster and be more flexible. I still see a lot of RHEL 3 and SLES 8 out in the field because it does just work, but I keep the newest OpenSuse, Fedora and Ubuntu going at home and the lab to get ready for the future.

One thing about being an open source administrator is that sales and marketing people will be coming in all the time pushing something. Free software has far less resources to push its solutions. In your role you'll have to offset the professional sales types with the real savings open source provides. The one advantage I've seen time and again on the Linux side is the ability to dramatically shift the admin to server ratio. I've seen industry studies that have the number of Windows admins to servers as low as one admin for every seven to eight servers, but I give the benifit of the doubt and make the ratio 1:50.

As a Linux admin you'll quickly discover that with proper planning and setup you can administer significantly higher ratios. Plan for proper provisioning, patch and configuration management, and monitoring you can hit 1:100 easily. I've seen clusters with 1:2000 ratios and a thin client setup that has 12K clients and 250 servers basically administered by two people.