Best Solution For HA and Network Load Balancing? 298
supaneko writes "I am working with a non-profit that will eventually host a massive online self-help archive and community (using FTP and HTTP services). We are expecting 1,000+ unique visitors / day. I know that having only one server to serve this number of people is not a great idea, so I began to look into clusters. After a bit of reading I determined that I am looking for high availability, in case of hardware fault, and network load balancing, which will allow the load to be shared among the two to six servers that we hope to purchase. What I have not been able to determine is the 'perfect' solution that would offer efficiency, ease-of-use, simple maintenance, enjoyable performance, and a notably better experience when compared to other setups. Reading about Windows 2003 Clustering makes the whole process sounds easy, while Linux and FreeBSD just seem overly complicated. But is this truly the case? What have you all done for clustering solutions that worked out well? What key features should I be aware for successful cluster setup (hubs, wiring, hardware, software, same servers across the board, etc.)?"
Pound (Score:4, Informative)
At work we have a pretty good experience with Pound - it's easy to set up & it load balances and will detect when one of your servers is down and stop sending traffic there. You can get hardware load balancing from people like F5 too.
If you're just starting out you'll probably want to start with software and then, if the load demands it, move to hardware
Machine-wise, we use cheap & not overly powerful 250 GBP, 1u servers with a RAID1; they'll die after a few years (but servers will need to be refreshed anyway) and they provide us with lots of options. They're all plugged into 2 gigabit switches
Some information about HA (Score:4, Informative)
I want to give you some more information. Based on your visitor estimates I think you do not have a lot of knowledge about it. Because for this number of visitors you do not really need a cluster.
But now to the other stuff. Yes, Windows clustering is (up to Win Server 2003 [1]) a lot easier. But this is because it is not really a cluster. The only thing you can do is having the software running on one server, then you stop it and start it on the new server. This is what Windows Cluster is doing for you. But you can not have the software running on both servers at the same time.
If you really want to have a cluster then you need probably some sort of shared storage (FibreChannel, iSCSI, etc.). Or you are going to use something like DRDB [2]. You will need something like this too if you want to have a real cluster on Windows.
I recommend you to read some more on the Linux HA website [3]. Then you get a better idea what components (shared storage, load balancer, etc.) you will need within your cluster.
If you only want high availability and not load balancing then I recommend you to not use Windows Cluster. Better set-up two VMware servers with one virtual machine and then copy a snapshot of your virtual machine every few hours over to the second machine.
[1] I don't know about Win Server 2008
[2] http://www.drbd.org/ [drbd.org]
[3] http://www.linux-ha.org/ [linux-ha.org]
Re:1000+ a day isn't very much (Score:5, Informative)
You'll need something that detects the primary server is offline and switches to the backup automatically. You might also want to have a separate database server that mirrors the primary DB if you're storing a lot of user content, plus a backup for it (though the backup DB server could always be the same physical machine as one of the backup webservers).
On this note, if you're comfortable (and your application is compatible) with Linux+Apache, then heartbeat [linux-ha.org] and DRBD [drbd.org] will do this and are relatively simple to get up and running. Just avoid trying to use the heartbeat v2-style config (for simplicity), make sure both the database and apache are controlled by heartbeat, and don't forget to put your DB on the DRBD-replicated disk (vastly simpler than trying to deal with DB-level replication, and more than adequate for such a low load).
Oh, and don't forget to keep regular backups of your DB somewhere else other than those two machines.
Re:1000+ a day isn't very much (Score:5, Informative)
I agree that 1000 unique visitors is peanuts, but as for how to do HA, it really depends a lot on your situation. For example, the primary server for Game! [wittyrpg.com] started acting up about 2 weeks ago, but it mattered little as I was able to flip over to the backup server and came out with barely any downtime and no data loss. In the mean time, I was able to diagnose and fix the primary server, then point the traffic back at it. In my case, all the dynamic data is in MySQL, which is replicated to the backup server, so when I switched over I simply swapped the slave and the master and redirected traffic at the backup server. You also have to consider the code, which you presumably make semi-frequent updates to. In my case, the code is stored in SVN and updated automagically on both the master and the slave simultaneously.
Having said all that, there's more to consider than just your own hardware when it comes to HA. What happens if your network connection goes down? In most cases, there's nothing you can do about it except twiddle your thumbs while you wait on hold with customer service. Redundant Internet connections are expensive due to the fact that you basically need to be in a big (and expensive) colocation facility to get it.
Also, how easy it is to have HA depends largely on how important writes are to your database (or filesystem). Does it matter if this comment doesn't make it to the live page for a couple seconds after I hit submit? No, not really. Does it matter if I change my equipment in Game! [wittyrpg.com] but don't see the changes immediately? Yes, definitely. Indeed, if your content is 100% static, you can just keep a dozen complete copies and put a load balancer in front that pulls dead machines out of the loop automagically and be done with it.
Nginx (Score:2, Informative)
For LoadBalancing and statics file HTTP serving use Nginx, is the fastest around. Use two or more linux servers for your High Availability Cluster, set a virtual IP for the LoadBalancer and HeartBeat to switch the virtual IP in case of failure. Software cost including OS = zero.
Amazon EC2 (Score:2, Informative)
Amazon's servers allow you to scale vertically and horizontally. They have images that are preconfigured to do load balancing and they have LAMP setups. Plus the fact that its a completely virtualized system means you never have to worry about hardware failures. with only 1k uniques per day, they have more than enough to accommodate for what you need
as for ease of use, i've never done windows load balancing, but the linux load balancing isn't terribly difficult to get working. to optimize it is quite a bit more difficult though. but with anything linux, its all terminal so its almost never as convenient as point and click. however, its almost always more flexible than point and click.
one other thing that you need to think about that goes hand in hand with HA systems is monitoring. with or without amazon, you need to always account for software failures too. apache might hang, the database might be overloaded, etc. you'll need something like nagios, cacti, etc. so don't forget to account for that in your hardware costs
HaProxy (Score:5, Informative)
Haproxy [1wt.eu] is better than Pound, IMO. It's lightweight, but handles immense load just as well as layer 3 load balancing (LVS), with the advantages of layer 5 proxying. It uses the latest Linux APIs (epoll, vmsplice) to reduce context switching and copying to a minimum. It has a nice, concise stats module. Its logs are terse yet complete. It redirects traffic to a working server if one is down / overloaded.
Re:Amazon EC2 (Score:1, Informative)
Perhaps you might want to take a look at this:
http://paste.lug.ro/75
It's an excerpt from a recent Undernet IRC session. So much for integrity and security on Amazon...
Re:1000+ a day isn't very much (Score:2, Informative)
Definitely. I had a site that was doing ~2000+ unique per day, used considerable bandwidth (lots of images). However, everything was heavily cached (no on-demand dynamic pages). And it was running on all on an old P4 and 512MB of RAM with fantastic response times and zero issues.
Pointless (Score:5, Informative)
1000 users a day? So what? That's less than one user a minute. Even if you assume they stay on the website for 20 or so minutes each, you're never looking at more than about 20 users at a time browsing content (there will be peaks and troughs, obviously). Now picture a computer that can only send out, say, 20 x 20 pages a minute (assuming you're visitors can visit a full page every 3 seconds) - we're talking "out of the Ark". Unless they are downloading about half a gig of video each, this is hardly a problem for a modern machine.
I do the technical side for a large website which sees nearly ten times that (as far as you can trust web stats) and it runs off an ordinary shared host in an ordinary mom-n-pop webhosting facility and doesn't cost anywhere near the Earth to run. We often ask for more disk space, we've never had to ask for more bandwidth, or more CPU, or got told off for killing their systems. Admittedly, we don't do a lot of dynamic or flashy content but this is an ordinary shared server which we pay for out of our own pockets (and it costs less than our ISP subscriptions for the year, and the Google ad's make more than enough to cover that even at 0.3% clickthrough). We don't have any other servers helping us keep that site online (we have cold-spares at other hosting facilities should something go wrong, but that's because we're highly pedantic, not because we need them or that our users would miss us) - one shared server does the PHP, MySQL, serves dozens of Gigabytes per month of content for the entire site, generates the statistics etc. and doesn't even take a hit. I could probably serve that website off my old Linux router over ADSL and I doubt many people would notice except at peak times because of the bandwidth.
Define "massive" too... this site I'm talking about does multiple dozens of Gigabytes of data transfer every month, and contains about 10Gb of data on the disk (our backup is now *three* DVD-R's... :-) ). That's *tiny* in terms of a lot of websites, but equally puts 99% of the websites out there to shame.
Clustering is for when you have more than two or three servers already and primitive load-balancing (i.e. databases on one machine, video/images on another, or even just encoding half the URL's with "server2.domain.com" etc.) can't cope. In your case, I'd just have a hot-spare at a host somewhere, if I thought I needed it, with the data rsync'd every half-hour or so. For such a tiny thing, I probably wouldn't worry about the "switchover" between systems (because it would be rare and the users probably don't give a damn) and would just use DNS updates if it came to it. If I was being *really* pedantic, I might colo a server or two in a rack somewhere with the capability for one to steal the other's IP address if necessary, or have DNS with two A records, but I'd have to have a damn good reason for spending that amount of money regularly. If I was hosting in-house and the bandwidth was "free", I'd do the same.
Seriously - this isn't cluster territory, unless you see those servers struggling heavily on their load. And if I saw that, I'd be more inclined to think the computers were just crap, the website was unnecessarily dynamic, or I had dozens-of-Gigabytes databases and tens or hundreds of thousands of daily visitors.
You're in "basic hosting" territory. I doubt you'd hit 1Gb/month traffic unless the data you're serving is large.
Do it the way google doe it :) (Score:1, Informative)
Buy 2 very cheap computers with double HDs. You can get them for less than 200$ each. Then install BSD/linux with mirrored raid. Then you can use rsync/unison/name your favorite synchronization tool to mirror data between computers.
Then use http://en.wikipedia.org/wiki/Lighttpd [wikipedia.org] or http://en.wikipedia.org/wiki/Nginx [wikipedia.org]. You will get relative easy setup, excellent performance, unbeatable stability and good load balancing that scales to 10k+ users in a hour.
Of course all is depend if you use bloatware or not. It is very easy to make dynamic content generation and database stop scalability to only few connections.
So all basic tools are easily available from any free server distribution.
IF YOU WANT 100% AVAILABILITY: Don't forget your networking stuff. You have to have 2 routers and 2 Internet connections. This is why server hosting companies are 10x better and cheaper than doing your own server.
From hosting company you get 24h administration and regular backups. And as a bonus you get pre-installed and pre-configured environment.
We will load test... (Score:2, Informative)
I see there are already a ton of good advice here, so when you have your kit set-up, post a link so that we can load test your config :-)
It's called the slashdot effect and if anything, you will at least know when things break and how your configuration handle these fail over conditions.
PS: This is cheaper then buying load testing kit and software :-)
Re:Is It Mission Critical? (Score:5, Informative)
And you're done. Oh, make sure to disable tcp_checksum_offloading on your webservers, else LVS won't work that well (read: not at all).
Just a heads-up for those who (like me) read this and thought: "WTF ? LVS works fine with TOE", it is a problem specific to running LVS in Xen VMs where the directors and realservers share the same Xen host. Link. [austintek.com]
Re:1000+ a day isn't very much (Score:5, Informative)
I was thinking along the same lines.
But to the person asking the question, if you want a full answer then you need to get your site built and make use of stress testing tools such as JMeter for Apache or Microsoft's WAS tool for IIS.
It's not something anyone here can give you a definite answer for without knowing how well your site is implemented and what it actually does.
Look into Transaction Cost Analysis, that's ultimately what you need here, a good start is this article:
http://technet.microsoft.com/en-us/commerceserver/bb608757.aspx [microsoft.com]
or this one:
http://msdn.microsoft.com/en-us/library/cc261632.aspx [microsoft.com]
Don't worry that these are MS articles on MS technologies they both still cover the ideas that are applicable elsewhere.
Even though no one here can give you a full answer for the above mentioned reasons, we can at least give you our best guesses and this is where I think the parent poster is spot on, 6 servers is absolute overkill for this kind of load requirements and indeed, unless your application does some pretty intensive processing I see little reason why a single server couldn't do the trick or at least a web/application server and a database server at most.
For ensuring high availability you may indeed need more servers of course and as you mention a requirement for FTP is bandwidth likely to be an issue?
The fact you're only expecting 1000 a day suggest you're not running the biggest of operations and although it's nice to do these things in house it may just be worth you using a hosting provider with an acceptable SLA, at the end of the day they have more experience, more hardware, more bandwidth and can probably even do things a fair bit cheaper than you can. Do you have a generator to allow continued provision of the service should your power fail for an extended period for example? If you receive an unexpected spike in traffic or a DDOS do you have the facility to cope with and resolve that like a big hosting company could?
There are many things I wouldn't ever use an external hosting provider for, but this doesn't sound like one of them.
Re:Some information about HA (Score:2, Informative)
The only thing you can do is having the software running on one server, then you stop it and start it on the new server. This is what Windows Cluster is doing for you.
That's not true. For clustering of front-end services (ie, IIS) you use NLB which is fully configurable load balancing and fault tolerance.
As already stated : HAProxy (Score:5, Informative)
Re:HaProxy (Score:3, Informative)
We get upwards of 15,000 hits per hour and just use Carp and Pound to handle our redundancy (Carp captures servers down, pound handles TCP ports going missing) across two machines (both RAID5 with FA RAM). Last time I checked the load averages, the 2.2 G processors were doing ~1.28 for a highly dynamic site.
Re:Is It Mission Critical? (Score:5, Informative)
If I had mod points, I'd give. This is the same thing we did, just different software.
-get 2 ISP, I suggest different transports. We have one as fiber, the other is a T1. There's no point in getting 2 T1 from different companies if a bulldozer cuts them together.
-Two dell 1950's
-Set each up with vmware server
-created 2 databases, replicating to each other
-Created 2 web servers, each pointing at database on same machine
-installed to copies of Hercules load balancer, vrrp + pen
-set up failover DNS with 5 minute expiration.
Now, you may say, why the load balancers if you're load balancing with DNS? Because if I have a hardware/power failure that's one instance where the 5 minutes for DNS to expire will not incure downtime for my customers. It also gives me the ability to take servers offline one at a time for maintenance/upgrades, again with no dowtime.
I have a pretty redundant setup here and the only thing I've paid for is the software.
Future plans are to move everything to Xenserver.
Re:Some information about HA (Score:3, Informative)
True, sorry I did not write it that clear. I was only writing about the Cluster software included with Windows. Not about other applications like NLB included with Windows too.
I just wanted to make clear that Microsoft Cluster Server is a lot easier to set-up (what the questioner has seen correctly) but this is because you get a lot less. He would have to install and configure several other applications (like NLB) to get the same as he gets with Linux HA.
CentOS/HA (Score:5, Informative)
It's fairly trivial to install RedHat/CentOS based clusters, especially for web serving purposes.
There are a few components involved:
1) A heartbeat to let each node know if the other goes out.
2) Some form of shared storage if you need to write to the filesystem.
3) Some methood of bringing up services when it fails over.
A web server with a backend database is one of the canonical examples. You'd install the heartbeaat service on both nodes. Next, install DRBD (distributed replicated block device). Finally, configure the services to bring up during a failure. The whole process takes about an hour following instructions on places like HOWTOFORGE.
But 1000 visitors a day is not much. It's small enough that you could consider virtualizing the nodes and just using virtualization failover.
Hire a technical architect (Score:2, Informative)
There are way to many questions that need to be known before a competent technical architect can help design the "just right" solution for you.
Most of the people here are experts on some small part of the solution and will spout "all you need is X" - and that's fine for free. I've worked on telecom - can never go down - systems for over 10 tens as a technical architect leading project teams from 1 to over 300 software developers and 20 others on the hardware side.
On the surface, FTP and web pages don't sound like the best solution to the problem as stated. Did yo just learn HTML and want to use it?
Now, here's my $0.02 on your problem: /. may still use pound for load balancing, so you know it scales.
* 1,000 visitors a day can be run from my cell phone. That's "nothing" traffic for a network or an old desktop.
* Avoid clustering at the OS or application level unless you really, really need it. You probably don't. Almost nobody needs clustering.
* Use network load balancing. There are many, many solutions for this. The easiest is from F5 (buy through Dell), but free versions work fine too - I've been using `pound` for years myself.
* Backups are key. RAID is not backups. Verify that you can actually **recover** from bare metal using your backups. Don't pull a Ma.gnolia http://blog.wired.com/business/2009/01/magnolia-suffer.html [wired.com]
* Disaster Recovery is important. Often, you can solve both backup and recovery and DR at the same time.
If you are a non-profit doing something I believe in, I'll do network, systems, B&R, and DR deigns and consult with you for free, an enterprise class solution. My company looks at FOSS solutions first, before recommending commercial, costly solutions. All our internal systems are FOSS, though we do have a lab with Microsoft servers since that's what many customers demand/need.
Think of a good TA just like a CPA or Lawyer. You pay us to prevent all the problems that could happen later that cost your huge amounts of money. After my CPA does my taxes, I sleep better at night.
Re:1000+ a day is trivial have you thought of amaz (Score:5, Informative)
HAproxy (which is the one I use) has the ability to define "backup" servers which can be used in the event of a complete failure of all servers in the pool, even if there is only one server in the main pool. If you're trying to do this on the cheap, that may help. It also has embedded builds for things like the NSLU2, so it may be easy to run on an embedded device you already have.
Use CARP (Score:3, Informative)
CARP is a protocol that does automatic load balancing and IP failover.
Install your application on 2 (or more) servers, give them the same address virtual IP address using CARP, et voila. Nothing more do buy, and no need to install any load balancer.
CARP's reference implementation is on OpenBSD, and it's shipped by default. DragonflyBSD, NetBSD and FreeBSD ship with an older version.
wrong on several counts... (Score:4, Informative)
Windows clustering allows for Active/Active clusters, so you CAN run the same service on two cluster nodes at the same time (with the exception of Exchange).
Setting up two servers to host VMWare guests and copying is not a good idea either - the HA tools for VMWare are expensive, and totally unneccessary for the proposed deployment. Without these HA tools, he would have to down his primary guest every time he wanted to make a snapshot.
We're talking about a very simple deployment here - HTTP and FTP. You don't even need clustering or a dedicated load balancer - instead, try using round-robin DNS records to do some simple load balancing, and then use a shared storage area as your FTP root (could be a DFS share for Windows or an NFS mount in Linux). This would give you a solid two-server solution that works well for what you're trying to accomplish, and adding servers would be trivial (just deploy more nodes, and add DNS records to the list).
If it grows much larger than 2 nodes, you might consider an inexpensive load-balancer; Barracuda sells one that works well and will detect a downed node.
Clustering for this job is totally unnecessary though. You're wasting your time by looking into it.
Re:1000+ a day is trivial have you thought of amaz (Score:5, Informative)
It's probably more complicated and overkill for what the poster needs, but it worked great for us. We used this years ago for transaction processing (~100,000 transactions an hour, not too busy) on a couple old HP NetServers with 1GB RAM each.
Re:1000+ a day isn't very much (Score:2, Informative)
I can attest to this. This is the same setup we used which had VoIP, DB, and HTTP. We ran OpenVZ on CentOS on DRBD. Each openvz virtual machine ran a service, sometimes several of the same services (db and voip). because fo DRBD, redundancy was taking care of, an using heartbeat, well, high availability was also easy. It worked very well. the only thing i would say, it is takes some knowledge and much elbow grease to get this working and plenty of testing. where as, some of the other solutions would be easer.
F5 is the choice (Score:2, Informative)
If you haven't looked at the F5 product line you should. The ability to use TCL language to write "iRules" and shear performance of even the smallest device is amazing. The devcentral.f5.com site is also great and allows you to gain from others experience. With an F5 in front the rest of the systems behind can be simple and cookie cutter with no complex setup. The F5 will handle persistence, load-balancing, and once you have your setup you can forget them for the most part.
For the FTP server part, you just need some Linux boxes running your favorite daemon and a shared storage for the files.
--russ
Re:1000+ a day isn't very much (Score:3, Informative)
To truly be HA you would need global load balancing. Your global load balancers are in essence the master name servers. You have 2 or more physical locations and the loadbalancer serves DNS lokups with very very low TTL to a site that is up (and if its more advanced to a site that is closest to the requester). If a DC blows up, your site stays up.
For each site dual ethernet drops, dual firewalls, and dual loadbalancers with a separate connection from each LB to server. Each piece of network gear has connections to both of the network pieces above and below it. (FW1 has access to both ethernet drops and access to both LBs etc)
Use hardware based firewalls and loadbalancers. Simple software based solutions do work, but their complexity, efficiency, performance and reliability cannot scale to what a dedicated ASIC piece of equipment can do.
Be mindful of power. Servers and network gear where applicable get dual power supplies fed from DIFFERENT legs. Do _not_ load circuits past their mid point. (Say both legs are at 75%. Leg 1 fails, all power is drawn from leg 2. It jumps to 120% or so and then it trips the breaker)
That being said 1000 unique visitors a day can easily be handled on most shared hosting platforms. If you're looking for uptime remember that you get what you pay for. And shared hosting is cheap.
Re:Is It Mission Critical? (Score:3, Informative)
VMs are like a bullet-proof vest for your hardware.
If a virtual machine takes it in the ass and crashes, the system can spawn a new one without missing a beat, whereas the same crash on the actual machine might cause it to crash.
It's also a good strategy to provide for future growth...If your machines are already virtual, you can host them on any hardware that's appropriate, and you can run as many as you need.
Buy an appliance (Score:3, Informative)
Your needs for 1000+ uniques are minimal. If I were to do it, I'd get a shared hosting account someplace and move on. Shared hosting can handle *way* more than that.
But if high availability (limited downtime) is part of the requirements, I'd say go out and buy an F5 BigIP. You plug your internet in the front, your machines in the built-in switch, configure your domain names in it using the web interface, and you're done. Set it to do service-checks, and it'll automatically pull out of the pool any machine that fails or that you take down for maintenance. So you get full up-time so long as your power and network don't fail.
Yes, you can get the same functionality using Linux HAProxy. But you sort of need to understand what you're doing. Reading the way your question is asked, I suspect you're learning this, and do you really want to make the mistakes on a real live project? Just go with the appliance until you have a solid understanding of what you're doing. Shoot -- I have a good solid understanding from years of experience, and I still use the BigIP when I have a budget (and HAProxy when I don't). It's just easier, and I can move on to more interesting problems with my time.
Once you've got this setup, set up a cron job to rdist the site to all the machines so that all your data is always on each machine. If you've got a database, you have some choices. For completely static data, I like to have it replicated to each machine, and have each web server just query localhost. If it's dynamic, have a replicated pair. At your levels, that can exist on the web servers.
I really dislike the cross-mounted disk architecture of traditional cluster solutions, because there are too many shared components. Each of those multiplies your possible points of failure for your whole setup. Better to keep everything completely separated, so if one component fails, that whole machine just drops out and the site keeps working because of the load balancer and because each machine can operate by itself.
Keep it simple (Score:3, Informative)
"I am working with a non-profit that will eventually host a massive online self-help archive and community (using FTP and HTTP services). We are expecting 1,000+ unique visitors / day. [...]"
Others have pointed this out to you, but 1,000 visitors is not much load at all. I work at a large university, and during registration first day of classes, we have 500 unique users (what you call "visitors") in each hour. On the first day of classes, we may get 1,000 unique users per hour as students look up their class schedules, and sign in to the registration system to drop that stupid class they were just in. We run a load balancer at the network level, so that traffic is balanced immediately at the switch, rather than at a host level before being sent to a back-end web host.
But doing the same in your case will be very expensive. If you work at a non-profit, you probably don't have this in your budget.
If you're just doing simple http and ftp (that is, not running a web application with a database back-end .. or an application that keeps "state" on the server, requiring users to always go back to the same server server they first visited) then you might consider the simplest solution of all: DNS round-robin. Simply put, you enter the IP addresses for two web servers (or ftp servers) for a single www entry in DNS. At the expense of hitting your DNS more frequently, you could set the TTL to 1 hour for the round-robin so that if server #1 went down, you could push an update to DNS so "www" just points to server #2, and users are only inconvenienced for about an hour.
But your best solution is probably just to outsource this, especially if you're only doing simple http and ftp. A good web hosting company already has this infrastructure available to you. No need to re-invent the wheel for just 1,000 users.
Rather than cluster, separate function (Score:2, Informative)
A) 1000 a day is fairly small. I server 12,000 unique logins per day with 1 web server (multiple back-ends, so point b)
B) Rather than cluster the entire application/site, it is usually better to separate the applications and processes and give them either their own virtualized server space or their own physical server.
Database on one server
Middleware/application on another
Static content on another, etc..
Not only can you figure out bottlenecks easier, but when/if you need to upgrade, you are putting resources directly where they are needed.
In terms of high availability, (in addition to the usual hardware duplications and backups/failovers, etc..) I would recommend virtualizing all your services into something like ZFS containers or vmware.
If a server dies, being able to quickly transfer a virtual zone (from backup) to a new server is very nice.
Re:1000+ a day isn't very much (Score:3, Informative)
Re:1000+ a day is trivial have you thought of amaz (Score:4, Informative)
2 boxes for hardware failover will do you fine, if you are worried about HA the its the COST of downtime that you are worried about (i.e. down for an hour exceeds $1000 in lost revenue) which will justify the solution. Don't just drive availability to five nines because you feel its cool, do it because the business requires it.
This is something that is rampant: techies tend to overestimate the value of uptime.
Sure, it's sexy to have high availability this and redundant that, but unless your company is pulling down at least $1,000,000 per year or more in gross revenues, it's hard to beat the 3 to 4 nines or so uptime delivered by a good quality, whitebox server running Linux. Something like this unit [aberdeeninc.com] would deliver excellent performance and excellent reliability at a very low cost.
How much does an hour of downtime actually cost your company? Be honest. If you had to tell your customers: "we were down for 2 hours because a software update caused us to have to ..." what would it actually cost your company? Especially if it only happened every year or so? In my experience, even in fairly stiff production environments, there has been no cost at all. We've maintained about 99.95% uptime for the past 3 years, with 1 "incident" every year or so, with no cost at all. In fact, our company has a good reputation for availability and support!
So don't spend money on sexy hardware with lots of blinkie lights and cross-connects, which often decrease your reliability by introducing unnecessary complexity.
Instead, spend money on your hosting. Don't *ever* host it in-house. Ever. Get a first-tier hosting facility, with redundant network feeds, power, and staff who give a damn. Don't be afraid to pay for it, because it will probably save you money, anyway. You'd be amazed at how price-competitive top-notch hosting farms can be!
Make sure to get to know the on-site techies on a first-name basis, give 'em a six-pack of their favorite beverage, and thank them profusely when they do anything for you. The goodwill these types of things can bring will work wonders for you down the road.
And remember:
2 nines is 3.65 days of downtime per year. .365 days of downtime per year (~ 8 hours) .0365 days of downtime per year (~ 45 minutes)
3 nines is
4 nines is
It's a very, very rare case indeed where 3-4 nines of uptime isn't completely sufficient.
And 1,000 unique visits per day? Pssht. Unless you are doing some pretty ferocious database stuff, (EG: joins across 12 tables with combined inner/outer/composite joins) the aforementioned server should do the job just wonderfully.
DON'T FORGET BACKUPS! And backup your backups, because backups fail, too.
Re:1000+ a day is trivial have you thought of amaz (Score:3, Informative)
http://www.barracudanetworks.com/ns/products/balancer_overview.php [barracudanetworks.com]
Re:Some information about HA (Score:3, Informative)
Please look at http://www.drbd.org/home/mirroring/ [drbd.org] and the next chapter "Recovery".
I hope hope this can help you already little.