Server Optimization For Newbies?

supaneko writes "I recently took a new job as a network and server administration for a small IT company. I am absolutely shocked at how much is taking place within this company that I have little to no experience with. To help bolster my experience, I purchased a used server to use for hands-on training and practice. My ultimate goal is to have a complete, secure LAMP server available to the public running CentOS. I have been browsing the Net for various guides and tips on setup, optimization, security, and maintenance, but nothing I've found really gives me a hands-on approach to the topics I want to learn about. When you all started out, what route did you take to pick up the server setup and maintenance skills you have now? Is there anything in particular that you would recommend to someone who has excellent skills with consumer PCs and servers but is a total newbie to corporate and enterprise networking and servers?"

Virtualization

Learn about virtualization. Take your pick of free offerings: ESXi and Virtual Server from VMWare, Xen, Virtual Server from Microsoft, etc.

Using virtual servers that are hosted on your new physical server will allow you to set up any kind of operating system you want and any applications on that operating system again and again and again with no fear of messing anything important up. Also, you can run (depending on memory) multiple operating systems side by side.

From there, you can start diving into learning all the operating system, application server, database server, etc minutia you like!

Oh, and don't forget learning about P2V. That will allow you to do all kinds of "what if" scenarios without affecting real servers.

Re:Virtualization

Virtualization is probably the greatest training technologies ever created, especially for the Network Administrator and Server Administrator.

Re:Virtualization

I disagree, strongly.

He already said he's using his own server for educational purposes. If he breaks something, he'll have to fix it.

We learn by doing, there is no other way.

Also, the virtual platform can be hard to set up and optimize itself, and can cause confusing or misleading stats from your platform's performance monitoring tools.

Re:Virtualization

Spend some time playing with vmware - I think you will be pleasantly surprised with just how close it is to running on the bare metal.

The only thing I don't use virtualization for is playing games that rely on frames per second - other than that, I honestly doubt you could tell the difference (and funny thing is - some things run FASTER - backup and recovery of the entire machine is as simple as copying some files from one hard drive (your backup set of vm files) to another. I can have a complete restore in about 5 minutes, and I can dupe a machine in about 6 minutes.)

Re:Virtualization

Plus, you don't optimize by adding another abstraction layer [...]

No, but you enable optimisation thereby.

In system design, abstraction is one of the best things you can do for performance, because it forces you to insulate your components from each other, and forces you to think about the interfaces through which they interact.

In an appropriately abstracted system, if you find a performance problem, you can then swap out a piece and swap in a new one, and everything should still work. Or you can move a virtual server onto a new physical server, and everything should still work.

Re:Virtualization

Virtualization is a wonderful learning tool. However, this being slashdot I am feeling a bit rantish.

Taking a job where you don't have any experience is fine when you have someone to learn from. However, having cleaned up my fair share of messes, or as I call them 'live learning environments'. I would suggest you start working for someone with experience AND play in a virtual environment.

Virtualization is the future but this career field is beyond the infantile stage of hiring someone with no experience and having them in charge of your business. Entry level admins aren't THAT expensive. What do I mean by that? Most IT workers can halt a business if not destroy it completely with less than a day's work. There is a certain working order to getting to know how to do things right. Do tech work, watching the seasoned admins do their job well and getting in on the front lines. When you have learned all you can from them, move on to a new business or move up where you are. Don't take someones business and brag about how good you are because you are too proud to take an entry level position. Then then call up /. crying because you are in over your head.

I mean good lord, the number of people in the last 6 months I have had to work with in forums because they didnt understand what FSMO roles were, or what a port was, or get this having to clean up a router because the idiot thought that /24 meant 1-24. (their router had been like that for almost a year).

My advice? Quit and take a job where you can learn from someone, check your ego and learn. All you are going to do by yourself is pick up a bunch of bad habits and a HUGE ego because no one is going to be there to tell you how much of an idiot you are being.

My Approach

I'm actually a software developer, but I work at a place with a lot of small projects and we do our own IT...meaning that we don't get budget for a dedicated IT staff and we end up doing a lot of it ourselves.

So, the way I learned what I need to know was to mess up a lot and get yelled at a lot. :-)

In all seriousness, we have finally landed at a place where we host and run our projects on Amazon's EC2. Some projects are even sophisticated enough now to leverage the EC2 platform and third-party services such as Rightscale for truly distributed cloud computing...but this isn't absolutely necessary if all you want is a place to run your production system. Best of all, since it's all virtualized so it's foolproof to learn new tech. When you're going to make significant changes you just save a snapshot of the current system, use it to start up a new instance off to the side, and screw it up any way you want to figure out a solution, and you can always easily revert to your previous snapshot if necessary. Just make sure you keep organized on which snapshots are configured with what, and be diligent about removing old snapshots that no longer have any purpose (again, purely organizational).

We've found in our business that the cost of doing this is vastly less than maintaining a rack of servers...so even though most projects don't leverage the cloud, we still benefit. (And of course there's room to grow into the cloud, which is also very beneficial.)

Get started by reading up on EC2, S3, and get the ElasticFox plugin for Firefox.

Re:Virtualization

It was not off topic, the guy is a noob and had the wrong title for his question. Read again:

When you all started out, what route did you take to pick up the server setup and maintenance skills you have now? Is there anything in particular that you would recommend to someone who has excellent skills with consumer PCs and servers but is a total newbie to corporate and enterprise networking and servers?

The guy is asking how to work with serving apps in general, he is light years from optimizing them. Like most noobs they post something not knowing what the hell they are doing, way over their heads, asking about something trying to be smart by saying I am trying to set up a PDC in server 2008, but cannot get my exchange 2007 running because it says I am getting a conflict with another IP. Reading things like this and the question in this forum make me shiver and want to scream because there are so many things wrong with the statements I barely know where to start. And in my example the guy was thinking there was just an IP problem, when in actuality "Can open, Worms everywhere".

Google

Back when I learned, Google was around. Turns out, it still is.

Most of the modern linux distributions have excellent package management. Most of them take care of 99% of the deploy "correctly" or "securely" issues. The only downside is that no two packages put everything in the same place on the local file system. But that's no big deal, especially if you compare/contrast to other distros.

Shoot, you can get an Ubuntu server installed as a VM in 15 minutes. (I don't see the need for dedicated server hardware, unless you're focusing on nuances of driver and hardware setup.)

Follow these steps:
1) Install base
2) Install app from package
3) Add custom content to package
4) Scan with the whole slew of freebie security scanning tools
5) Realize that at this point, you're better than most orgs already.

O'Reilly Cookbook series are very hands on

If you want a more hands on, how do I accomplish a specific task type approach to things, I've been very happy with the books in the O'Reilly Cookbook [oreilly.com] line. They usually run 35-50 bucks depending on topic and you'll want to page through one in a store before purchasing. All the information in the books can be found online, but they usually organize them nicely in the books. Most of the topics are 1-2 pages responding to a specific "How do I do X" type question. The Linux Networking Cookbook, bash cookbook, and Linux Cookbook and Linux Security cookbook might be a good set to start with for what you are currently playing with.

One question

How did you get a job as a company's sole "network and server administration" (sic) when you are a "total newbie to corporate and enterprise networking and servers"?

In every case I've experienced where someone was hired for a sysadmin job with absolutely no experience, there was a more senior person on staff there to mentor/train them. But it doesn't sound like that's the case here.

So... either (a) you were completely up front with your employer about your lack of experience and they hired you anyway, in which case there's no problem because they have limited needs, know you're learning and don't expect much; or (b) you lied to them, in which case the answer is "quit and go get a job you're actually qualified for".

Re:One question

You forgot c) they fired the mentor with the junior barely trained and now the junior has to do the whole job by himself

Happens a lot more than you think

Re:One question

Or... He listed his experience, and the potential employer just nodded and pretended it meant something to them.

Optimization

Optimization is about finding bottlenecks and then using the scientific method.

The typical bottlenecks are CPU, RAM, Disk, and Network. A little research will reveal the tools that give you insight into those subsystems on your platform.

Using those tools, you can identify which processes are stressing each subsystem. Then a little more research will reveal the tools that give you insight into that process.

Then a little-to-a-lot more research will reveal what you can do to reduce the stress or beef-up your platform.

After you do this for a bit, you'll see why LAMP is usually referred to as a stack, and not as a turn-key server. Different parts of the stack need to be optimized for different subsystems.

Another very useful bit of research would be finding or writing your own tools to stress each of the subsystems.

The FreeBSD Handbook

The FreeBSD HandBook [freebsd.org] and a FreeBSD install cd.

Read-it end to end. Yes, i know it's huge. You won't regret spending the time to read it. Install FreeBSD (even in a VM) and use it. Even if you'll use other operating systems in the furture it's a good read and you'll learn a lot.

Mean bastards, aren't we?

Today is one of those days that I wish I had mod points.

First, the question at hand, get yourself some virtualization, and get a box that you can just plug in at home and fiddle around with when you aren't doing anything else. Trial and error will help you.

Just make sure that you do your trials and errors on a testing environment and not in production. It is alright to make mistakes until you sort stuff out, just don't bring down the house.

Second, shame on you naysayers. Let this guy learn stuff as he goes. Where did our curiosity and creativity go? You could give him advice instead of being a rude, mean, naysaying bastard. Thanks for posting as anonymous cowards too. Real nice.

Re:Mean bastards, aren't we?

I'd recommend the notebook approach, but I prefer to use a wiki. There's less chance of it being destroyed ... because the first thing you learned was how to make backups wasn't it.

A Wiki is better because:

you can cut and paste commands into it without errors - including urls
you can always read what you type into it
you will never spill coffee over it
age will never destroy it
you will never lose it in the office moves
you can share it with your colleagues
it will always be there when you're doing things at your computer (assuming you work with LAMP)
you can upload zips of config files, packages, etc

Whilst you could store passwords on it, I'd recommend against doing that :) a notebook (or keepass) is much better for them.

Some Words

While many other posters give you heat for not being knowledgeable, I commend you for making the effort to learn. Keep that attitude, and you will eventually get good at it!

As for optimization, my advice to you is:

1. Know what you need to optimize
2. Measure, don't guess

It's good to read some generic "how to optimize foo" advice, but be careful you don't end up spending your time and effort optimizing something that doesn't need it. Know what things need to be fast, and focus on those. Very often, you will find that, actually, everything is fast enough, which means you don't need to optimize anything at all.

Once you have determined what, if anything, needs optimizing (by measuring, of course), the main thing is to identify the bottlenecks. If your pages take a long time to render, is it the web server that's slow, the network connection, the web browser, the code on the page, the code that generates the page, the database, the filesystem, or something else? Once you know where the slowdown is, find out what's causing it. Again, measure, don't guess.

Then, once you know the cause, you need to think about how you can solve it. In many cases, this will be clear to someone who is skilled at working with whatever technology it is. For example, a good programmer will know how to improve a program, a good DBA will know how to optimize database access, etc. In some cases, however, you will find that the performance at your bottleneck can't be improved significantly. You can have a skilled programmer spend a couple of days to squeeze the last few percents of performance out of some function, but that isn't going to help if you need things to go twice as fast. In that case, you may be able to solve the problem by using more hardware or faster hardware, or you may simply not be able to solve the problem.

Optimization is less important...

First off, Optimization is less important.

You can spend days, week, or even longer... trying to make your systems run better and with fewer problems... but problems will crop up. And if you spent all that time just "Optimizing," you might find yourself between a rock and a hard place...

I learned early on that Backups are ever so important. Our shop doesn't do tape backups, but we do Disk-to-Disk backups of our virtual machines, and the backups are off-site. We also do a traditional file backup as well, with versioning.

Depending on your shop, money may or may not be an issue. Whatever you want to do, it can be done for every budget. The cheaper ways just require more time/expertise on your part, and that means it might not pass the "Mack Truck Test*." If your company wants something somebody else can step in with a basic training of how things work, you'll have to go with a more expensive solution.

Once everything is working like it should, then start working on improving it.

--Pathway

*: The Mack Truck Test - If a system requires some expertise to operate, and the sysadmin is hit by a Mack Truck, how long will it take for somebody else to fill the role of sysadmin? If the amount of time is acceptable to the employer, then it passes the Mack Truck Test.

Slashdot it.

Post a link to your server on Slashdot. I guarantee you'll get a fast and furious lesson in server optimization and security.

You're headed the right way.

You're headed the right way. Just keep going. I'd recommend Debian over CentOS, because it's the generic professionals distro, but that's not that important.
If you're feeling overwelmed by what is required to get a webstack up and running, you're absolutely right in that respect - its a non-trivial amount of stuff. Allthough it is a tag irresponsible to take such a job without the basic knowlege, mind you.

The classic LAMP webstack is solid but has lot's of components. Start with making a list of what you *don't* know, but would like to know. Formulate these out in questions and sidenotes to your self and write then down in a simple indented list in a text editor. Notch them of as you go deeper into each issue throughout the next few weeks.

Here's a list of things from the top of my head you need to know your way around as a professional admin:

- daemons on Linux/Unix

- cron-jobs

- the cli/Bash

- cli tools: wget, mc, emacs, ssh, scp, sort, ls, less, the concept of piping, rm, chmod, chgrp (these two will help you FUBAR your LAMP-stack a few times before you get a hang of it. Don't worry, we've all been there. :-) )

- learn VI or Emacs (the "No X" versions!!!). Get a book/download the docs/print out the cheatsheets. I personally recommend Emacs. Start today. Either are a pain in the ass and you won't bare any of those longer than 2 minutes in the beginning - their handling is bizar beyond any words - but 6 weeks from now, when you know your way about the 20 basic editing actions in Emacs and are logged in via SSH and have to digg through a script or a huge Apache config you'll be very thankfull.

- Learn Apache. Start with 2.2. Get a book. Oreilly is a safe bet.

- If the P in LAMP is PHP, learn PHP and do your maintenance scripting with the CLI version of PHP, thats what I do. Copying, maintenance, cron-jobs ... all in PHP. Very neat. You swat two flies in one move, as you can look into PHP code at app-level and find your way around should that be needed in an emergency.

- Replace PHP in the above paragraph with Python or Perl if required. If Emacs is your choice of CLI editor, Elisp is a good choice for scripting aswell.

- try to understand the file system and directory standard of Linux before you implement your own little world. A lot of the dirtree in Linux is a historically grown mess and up to individual disposition, but the essential security related stuff is not(!!). So don't mess around. Plan ahead. Take notes (on paper!) and be prepared for a reinstall after a week or two when you've totally borked your system or your systems rights.

- Learn a versioning system. I recommend SVN, as the newest hype, Git, is still to unwieldy to handle in most cases (not enough tried and true 3rd party tools). Learn the CLI of your versioning system and use it too, so you get a hang of it. Put your docs, custom configs and other files like scripts into versioning and use it. I strongly recommend "Pragmatic Version Control with [fill in favorite vcs here]" from the Pragmatic Programmers Bookshelf guys. Real world versioning without the useless theoretical bullcrap. A very good line of books that finally made me understand versioning the way it was meant to be. AND USE VERSIONING! F*CKING VERSION YOUR SHIT. At every occasion. I'm dead serious. Learn to use revert, diff, etc. DO PRACTICE IT! It seperates the pros from the wannabees. You'll eventually find out why. Trust me on this one.

- MySQL. Well, it sucks just as much as any other SQL RDBMS. If you hate SQL and all that comes with it with your mind, soul and body like I do, you'll just have to bite the bullet. Get a book with a good index and keep it around for hard times. Play with a few basics of the mysql cli client so you can get up to speed when you are in a jam. Don't waste to much time with it though. It takes a strange state of mind to deal with this kind of stuff. I've never quite gotten the hang of it. A GUI-tool can take the pain out of DB admining.

You need a good dose of common sense

Which, unfortunately, isn't that common.

Experience is the best teacher, but unfortunately it's not a particularly fast one. Anyone on /. can point you at a few interesting things like Slackware, Google and O'Reilly's back catalogue, and plenty of people already have.

What I would advise is:

1. Learn to see past the bullshit. There's a lot of it in IT, generally being spewed by salesmen and managers who pretend they know more than they do. In my experience, the less intelligible the communication (ie. the more buzzwords), the more likely it is you're talking to someone who doesn't have a clue. The word "Enterprise" is a good barometer there - it's often used completely unnecessarily and in the IT world has almost zero meaning.

Example: A Dell 2950 with every component that can be made redundant made redundant isn't an "Enterprise Server". It's a server. If you haven't specced it with redundant power supplies and disks, I wouldn't even class it as a server. It's a PC in a very expensive case.

2. Sometimes it's worth paying for a solution. /. would have you believe that Open Source is the Answer to All Our Prayers, and that Richard Stallman is the Messiah. Not true - there are plenty of products which don't have a half-decent open source alternative. Courier is a great IMAP server but at the end of the day, Exchange is a very capable product and is fantastically hard to beat feature-wise. Zimbra comes close but who knows what kind of a future it's got as it's owned by Yahoo. And I defy you to find a F/OSS business accounts system which isn't half-arsed. You can't say to the tax authorities "Errr... about those accounts we're due to submit - yeah, we just realised that our accounts system hasn't been updated to account for the recent changes in tax law and so we're having to wait until it is. Don't know how long that will take".

3. Security, security, security. Understand the ideas rather than just mindlessly installing the patches - a hardened Apache installation with a locked down PHP configuration behind a firewall operating some fancy layer 7 intrusion prevention system is great, and will help mitigate many forms of attack - but at the end of the day if you've got a badly designed PHP application all that'll happen is that intruders will access your data through a pretty web-based user interface.

4. Look at what the business does right now, think of how things could be made better and put together a system to make things better. It doesn't necessarily have to be something that will see the light of day - it could just be feasibility checking - but it'll give you something useful to do with definite goals which will teach you a great deal and at the same time may very well benefit the business.

Re:Slackware

Assembler. Forget about Slackware and all the other already-coded distributions. Learn assembler and code everything yourself. It's the only way to learn.

Re:Slackware

Nah, that's too extreme. Everyone knows that the best way to learn Unix is to run Gentoo [funroll-loops.info].

Re:Slackware

Slackware. Forget about Redhat and all the other GUI-fied distributions. Install Slackware and do everything yourself. It's the only way to learn.

This is good advice.

Actually, it's not very good advice.

Last I checked, Red Hat/Fedora/CentOS all have the exact same command line as every other distribution, and the system is configured using the same text files that have been used for nearly 20 years. All the GUI tools do is modify the config files.

For a newbie, having the GUI there to change a config then looking at what text file got changed (and how it got changed) is an excellent learning tool.

Also, last I looked, Slackware isn't one of the distributions that make a good bullet point on a resume. Red Hat, CentOS, and SUSE are good for real-world server skills, while Ubuntu, Debian, and maybe some Fedora would be good for Linux desktop skills.