Mys med kissen

foboasjldfkj

bar-foo

This blog is now subject to be my testing environment mohahahaaha

So our new servers were installed with Linux Fedora 11. This is a GUI based Linux, keep that in mind. The reason I installed a GUI based linux was because when you get to the servers in the server halls and you find a problem which you cannot handle. Not having your own laptop with you could prove fatal because you will have a hard time googling for answers in lynx. So having a GUI on the servers could save you there. Also, some future admins may not be so comfortable with CLI as I am, and therefore will appreciate the GUI for what it is.

But, on to the problem. We have a server pair which are simply two identical servers set up so for redundancy. One can be taken down for maintenance and the other pulls both servers loads. Problem was, when the one server which we can call server alpha went down or even just did a service network restart, the other server, which we can call server gamma, would jump and restart its network as well. This is not good of course, but what is even worse is that when server gamma restarted its network, the routings would go haywire and even though none of the interfaces had IP-addresses in the 192.168. range assigned on any network card, the default route would appear in the routing table as 192.168.1.1.

So, what I did was I used the route command to correct the routes, thinking they would stay the way I changed them. But after a service network restart to test it, the same results occurred. So I thought I had some bogus network script somewhere maybe. I pulled up a grep -r “192.168.1.” /etc/* but it yielded nothing.

I just couldn’t figure out what was wrong. I started writing route-eth0 and route-eth1 files to force my routes into the routing table. And this did just that. But it did not remove the wrongful default route to 192.168.1.1.

So after consulting with some Linux people on IRC I found out that Fedora does not leave the network handling to the conf files in /etc and the network service. Fedora has a service called NetworkManager. I shivered as I see saw this service-name in camel-casing. The dread which struck me when I saw it had to do with networking was even worse. This NetworkManager is a graphical tray-daemon which simply ignores all config files and holds on to what you told it the first time you started the server. Especially the default route it seems. After killing NetworkManager, the right configs were read by the network service and everything turned out right.

How to kill the NetworkManager.

First we need to make sure it does not start automatically on restart.

chkconfig –level 2345 NetworkManager off

We also want to make sure the network service does start automatically. Otherwise we will be without network.

chkconfig –level 2345 network on

Now we need to stop both the network service (if active) and the NetworkManager service to then just start the network service.

service network stop && service NetworkManager stop && service network start

I hope this helps someone else who runs into the same problem that I did.

So, now the servers have been put into place at our new co-location. The old servers have been replaced to full extent and have been moved from our old co-location to the new one aswell.

I thought I would tell you a bit about what problems I encountered during this server migration and how I came to terms with them.

The first and most obvious problem was, since we could not put in the new servers at our old co-location, how would we sync our 180GB databases so they contained the same data on the new servers as on the old servers, virtually in real time. Setting up db replication slaves would put too much of a strain on the system and would build up massive binlogs. Since the old systems were being replaced because of lack of disk space, this was not an option. I had to analyse the applications running on the servers and device a solution which only sent the core data and let the rest of the data be built parallel and identically on both systems.

So after many log debug sessions and consulting with the other developers who knew this system a lot better than I did, I realized that we had three forms of raw input into the database and the rest of the data was a result of that input.

One of the input forms came in as packages, once per day and customer at a total of about 300MB per day. All of these packages were automatically uploaded to our web server and saved in the appropriate place by the script receiving the file.

We solved this by modifying the script to curl each file which was uploaded, to the new servers. Thus when a customer uploaded their daily package to us, our servers automatically replicated that request to go to the new servers as well.

Another input form also came as packages but they came via email, once per day and customer. This was a no brainer, we set up the email address receiving the mail to simply forward all the mails to the new server as well.

The third input form was a bit trickier. Live data not sent by our customers but by their individual browser clients. Now the critical difference here is that, while I can take a snapshot of the databases from the old servers and transfer it during the day to the new servers and put it into their databases, that process took 18 hours. This would mean that any database I would put into the system would be up to 18 hours old, thus it would be missing 18 hours of this live data. This would not be acceptable. However. This is what I started by doing. We also put in place just like with the first input format, a curl which would transfer all the data as it came in to the old servers, from the old servers to the new servers.

So now I was left with an accurate database except there was an 18 hour gap in the data. Luckily, all the data from this input form was inserted into a single table of the database, and even if some customers had 6GB of data in this table, very little were used each day and much of it could be discarded. So since I previously set up backup scripts for the old servers as explained in this blog entry, I could start truncating this table so that it only contained as little as two days data. This meant that the replication of this table could go a lot faster.

By making the affected tables very small by truncating them and by opening the firewall on the old servers to let in database connections from the new servers I could sync each table for each customer individually and close the 18 hour gap completely. This, without putting too much stress on the old servers.

Having two sets of servers which did the same things parallel gave me a chance to clean up the old system as well. There were a lot of old artefacts of test code here and there, databases which were not used and files which simply could not be moved while the servers were in production. This opportunity gave me the chance to clean up after so much that had gone wrong previously. The new servers were not just a replica of the old servers. They were a clean and augmented version of the old servers.

After the servers were totally in sync and I was sure that they processed all data correctly and presented it correctly, I changed the NS records for our domain from the old server ip to the new server ip and let the transition slowly take place.

How did we know everything was identical on both servers? I put together a script which did the bulk of it. The script made two database connections, one to the new servers and one to the old. For each database, each table and each row, it made an md5 checksum and compared these checksums from both databases. I even did this without sorting the mysql result sets. When databases are identical, they really are identical it seems. The script gave out warnings for some of the tables, but all of those warnings turned out to be simply sorting differences. They were few enough that we could check them by hand. The script ran for approx 8h. I feel I could have made it more efficient, but it sufficed for our purpose.

Now with this server migration complete, I hope to be able to write a little bit more backend/frontend blogs here.

The server is done, installed and ready to spring into action! But you need some place to put it. It’s really uncomfortable to have it in your cubicle. Not to mention noisy. So you find a few companies who can offer you co-location services. This is where the problems start. You all of a sudden have to grasp the concept of bandwidth usage according to 95/5 principles and power needs of your server. What an SLA is and what it’s worth to you and what you physically need to put the server into a co-location.

Lets start with the hardest part (according to me).

Bandwidth costs according to the 95/5 principle.

Your co-location partner may say something like: “You can buy 2Mbit of me according to the 95/5 principle for €150 per month.” I should start by saying that this is an example price and should not be used in any way as a reference. It is a price I have seen, but prices vary too much for it to be any kind of reference.
But what does it mean when your co-location partner says “2Mbit” in this case? You will probably think you will need at least 10Mbit or perhaps even 100Mbit in order for the server to run smoothly. Because if your customers can not download their report PDFs fast enough, they are going to start complaining. This is where it gets interesting. Most commonly, the bandwidth you buy from your co-location partner is referred to as ‘Transfer’. This is the traffic that is transferred to your servers, not the traffic sent by them. This is something you should ask your co-location partner, of course. If you only pay for “transfer” or if you pay for all traffic. Now, which ever the bandwidth you buy is concerned, if it is calculated according to the 95/5 principle, there are two values you need to know. How many Mbit you buy, and what your burst capability is. You see, the burst is the actual line speed. How fast is the actual pipeline which you shovel traffic trough. This is burst. You can burst all the time if you want, but you will pay for it afterwards. The Mbit you buy is not guaranteed per default that you can use 24/7. This is where SLAs come in, but we will talk about them later. So, if the burst is 100Mbit, what does the “2Mbit” in the statement above mean?
Calculating bandwidth usage according to the 95/5 principle is done by excluding 5% of all time for the measuring of bandwidth, the 5% which is excluded is the 5% where the most bandwidth is used. After the exclusion of this, the highest Mbit load during any 5 minute interval, is the result. This result is what you pay for. In the case above, 2Mbit.
Important to ask is what the cost is for using more bandwidth, than you originally buy. If the price is the same per Mbit as what you pay otherwise, you may as well buy 0Mbit, since what you buy is just a base and you should have a better price for paying a stable amount each month.

Power needs of a server

Well, the power needs of servers are hardly ever easy to calculate. Even if you can do it quite easily by summing-up the Watt usage of each component in the server, the actual usage is up to the load of the server and the efficiency of the cooling at the co-location. But if you do manage to calculate the individual parts’ Watt usage and compare to other common server parts, you should be able to see if your server has a lower power usage than the other common servers. This means you should try to have your power price based on the actual power price paid by your co-location partner. If your server is a power monster however, for example servers with extreme RAM amounts of non low power RAM, or lots of high RPM HDDs. Try to get some kind of generic price per server from your co-location provider. These types of servers consume a lot more power with higher loads than regular servers do. So if you do not want a shock when the power bill comes, keep to a set price, if possible.

What is an SLA?

Service Level/Legal Agreement. This is a contract specifying the minimum level of service that the your co-location partner must be able to deliver. For example, how often may the network be down, or how often may there be a power failure. If the level of service on the SLA is not met, the customer (that is you) will be compensated. This type of agreement is very popular in countries where lawsuits are popular. In countries where people do not care about lawsuits, there is often no point in signing an SLA either. Remember that an SLA is not a guarantee that you will have a certain level of service for network capacity or power failures, it is simply a contract saying there is a set price for compensation from your co-location provider to you if they should fail. What you should look for in an SLA is that the actual amount you will be compensated for your co-location partner failing to meet the level of service specified, is enough to cover your cost of recovery. This means that if your power fails and you servers reboots, make sure things start manually and you have minimal work needed to be done in case this happens. This is a far more profitable solution to focus on than relying on an SLA to solve these problems.

The physical aspects.

Co-location partners have server racks in which you rack in your server. You often have to do this yourself. No one wants to be the one to drop the server, so it is your back which goes into it. Most servers have rails which you put into the server rack prior to putting the server in. This makes the installing very easy. Few servers are installed without rails and some cannot be installed without rails. Depending on what you buy from your co-location partner, you may have to have rails on your server because the chance of you getting the bottom of a rack is only 1/42 chance. Which means you cannot simply place the server on the “ground”. Some rail mounts are really easy to install once you read the manual for them, some require lots of work, patience and dexterity. What I want to stress is DO read the manual. You do not want your servers falling down.
Worth noting is that most server racks have two power outlets, one on each side of the rack. The one is often main power, and the other is thus UPS power. If your server has two power supplies, plug in one in each side. If your server has only one power supply, make sure to know which side is the UPS. You want the server in the UPS. Ask your co-location partner if it is not obvious which side is the UPS.
When it comes to power cables, some co-location providers use the conventional “computer cable”. However, this is not the standard in server racks. Ask your co-location partner about what kind of cable you will need. It may seem like a weird question, but you need to make sure which one is used. The most commonly used in server racks is the kind of cable which in the old days connected computer PSU and computer screen, a computer power extension cord if you want to call it that.

Good to think of when you start looking for co-location partners is to find a partner that fits your needs. Some co-location partners are very strict on what comes into their server racks and who goes in and out of the server park. Others are not. Ask yourself what level of security does your customers require you to have, and what level of security you feel is necessary. Most co-locations have redundant power and internet connections. Ask yourself if you need it now, and if you will ever need it. But most important of all. Make sure you do not bind yourself on a long term contract with a co-location partner that do not seem to be for you. Things can get ugly.

Last but not least. Make sure that your servers have someone close by who takes care of them. Cameras and monitoring clients are good, but nothing beats the usefulness of having actual people close to the servers at least during daytime.

I approached this at a different angle.

As developers we needed a staging environment for the application we are developing. Using the live servers is not okay in my opinion. So I found an old P4 box with 1GB ram. Slapped in a new HDD for the databases and started setting this up as a development server. Of course, porting the code to make it mobile enough to move it to this server was one thing, which is another story. But getting concurrent versions of the databases onto this server was were the backups came into play.

Using the development server as a test case for the backups, I had everything I needed to build a true case where I could assure that each and every backup taken would work. Each backup is restored to the development server and as we developers use the development server, we see if some data goes terribly wrong. Of course, this does not protect us from the scenario where some data is malformed in a uniformed way, producing false positives. However, all data which goes into our system is backed up separately, so everything can be rebuilt if needed. This data is checked via md5 hashes, so no malformation is possible there.

So, the backups are taken to a backup server, custom built for the purpose. This server is an office version. We wanted an out-of-the-server-park backup, so we decided to put a server at our office. The box is custom built by myself and based on a Chieftec chassis with an Intel motherboard and Intel processor. I used a highpoint rocketraid 3210 as a RAID controller. I built a RAID5 from four Western Digital 1TB hard drives. We also have two spares in the server, so in case we need to rebuild the RAID we do not have to wait for a delivery.

A word or two on security.

When doing cross server backup transfers, you must always make sure you know your security priorities. In my case I trust the backup server far more than the live servers. This is often how the case should be in my opinion. The backup servers are exposed to no one but the live servers and the developers. and the live servers are exposed to the whole world. Thus, the more likely first target of an attack would be the live servers. If the backup servers trust the live servers, and the live servers are compromised, the backup server will be compromised as well. This is why we build the backup scripts on the backup server and make them fetch the data from the live servers, rather than having the live servers push the data to the backup server.

The backup scripts are in our case built in php, so all developers at this  company can understand them. I would rather have built them in straight bash or perhaps python, but the advantage of more people being able to understand all code is far greater than having used “that cool programming language”. Of course the php executes complex shell commands to do most of the actual work, but the flow of the program is entirely php.

So, how do you set all this up?

The way to set up these kind of scripts are with ssh certificates, crontab and some basic php code.

First you need to set up a user on the live servers who will have only the privileges of reading the files/databases which needs to be backed up. I’m calling this user ‘backup’ for easy reference.
the adduser command is slightly different in many distros, so read up on it if you are unsure.

man adduser

Secondly you need to give the backup user on your live server the public key of the backup user on your backup server. So log on to your backup server as your backup user and type:

ssh-keygen -t rsa

a file “id_rsa.pub” should be generated in the .ssh directory in your home directory (~/.ssh/id_rsa.pub). This file contains one(1) line and this line is this user at this machines public key. It identifies this user at this machine. What you need to do now is to transfer this line into the live servers’ authorized keys file. (~/.ssh/authorized_keys). This file may not even exist on the live servers, but if it does, make sure only to add your public key line to the end of this file. Do not overwrite. The magical unix command for doing this would be:

cat ~/.ssh/id_rsa.pub | ssh backup@your.liveserver “cat >> ~/.ssh/authorized_keys”

of course, you should double check these kind of things manually and not rely on a copy-pasted command from blogs like mine. The command above does give you a hint on how the backups are transferred though.
After you have put the backup user of the backup servers public key into the authorized keys file of the backup user on the live server, you should now be able to log in from the backup server as the backup user on the live server as the backup user, without being prompted for a password. If this is not the case, you’ve done something wrong. Start from the beginning.

So, now we need to start writing the actual script which controls the flow of the backup process. I will now write an explanation which can be applied into any kind of programming language.

First thing we need to do is to get the information on what databases we need to back up from the server. We do this with an easy shell command executed from within the script, and then parsing the output it gives us. Why do we not use a local mysql client implementation to query the database directly and get a mysql resultset? Simple. We do not want to expose the live mysql to the outside world. So what do is we issue the following command in a shell controlled by the script (so we can get the output into our code and use it):

ssh backup@your.liveserver mysqlshow

This will initiate a new ssh connection to your live server, and the command mysqlshow is sent over that connection, just as if you would have logged in and done it yourself. All output from mysql will be returned and will end up in your script.
Once you have gotten this down you will have to parse what mysql throws at you. This should be a no brainer. I use this regular expression /[\w]+/ because all we really need to do is remove all spaces and pipes. Of course, this regular expression must match all your databases you need to backup. Maybe you only have one database to back up, in which case, you do not need to do this step really.

Next thing in the script is to, for each database we want to backup, get the dump from the database. But how do we do this without actually writing everything to disk as dump files? Easy. This command may seem complex, but I will walk you trough it step by step:

ssh backup@your.liveserver “mysqldump databasename | gzip -c” > /your_backup_location/databasename_date.mysqldump.gz

ok. this may look scary for some of you. If you are not familiar with the mysqldump command, what it does is it outputs in plaintext a full recreation script for the database in question. So we initiate an ssh connection and tell the server to start outputting the whole database. Then we take all which it outputs and hand it over to gzip, which zips the output and with the -c parameter it just passes the zipped output on. All of this ends up on your side of the ssh connection, where you in turn tell it to go into the file /your_backup_location/databasename_date.mysqldump.gz
All data is zipped and done when it gets to your side of the connection.
Restoring from this file is also easy. The following command can be used for that:

cat /your_backup_location/databasename_date.mysqldump.gz | ssh backup@your.liveserver “gunzip -c | mysql databasename”

So, having the script done we need to put this into a crontab on the backup server. Note that nothing except for the authorized_keys have been changed on the live server.
crontab -e
lets you edit your crontab. Here you need to write a crontab line. The crontab may be empty, in which case it is hard to remember which order everything should be put, so here is a reference:

min     hour     day_of_month     month      day_of_week      command to be executed

So if we want this script to be executed every evening at 23:42 except for on Sundays, because then the server does other heavy load things, we would write something like this:

42      23         *           *            1-6        /path/to/script

A better reference for crontab can be found here: http://www.adminschoice.com/docs/crontab.htm#Crontab%20file

Two identical servers, set up this way to make the system which they are housing, redundant. The servers are replacements for the currently used servers which are some old DELL boxes with dual core 1,8GHz processor and 4GB of DDR2 RAM each. The HDDs in these boxes are over-dimensioned cost wise. Four 15k RPM Ultra-SCSI 320 at 250GB each strapped as RAID 0+1 with a RAID controller that has 256MB of on-board memory. Doing redundancy at operative level is really important, but doubling the redundancy with two servers and mirrored HDDs like this just doesn’t give as much more real redundancy as it cuts into performance. And for the bill attached to the RAID, you could have built four servers instead. I’ll try to look up my source for the performance of two RAID1 disks vs four RAID1 disks. No link atm.

The replacement servers are a pair of HP DL160 G6 as a base. I added some ram on them giving them 8GB total each. The G6 house some nice DDR3 RAM which is perfect for SQL servers and allow for a total above 100GB of RAM per server. Sadly the database the servers have to house in the end is approx 180GB so i cannot load it all into RAM. I am really excited about building an NDB cluster some day. I am going to be gunning for it in some future investment.
The servers are fitted with two seagate 450GB 15k RPM SAS HDDs strapped as RAID 1 by a HP SmartArray P410 controller. All this works fine together.
The sad thing for me as a FreeBSD fanatic about the HP DL160 G6 servers and most of the G6 it seems is their network interface card. There are no drivers for it in any BSD system and therefore i choose to use Fedora. I am personally more biased towards Debian but i figured the GUI may give future admins of the server an easier time to administrate it. Choosing between personal biases and professional advantages is a no brainer.
The system is installed on a regular SATA disk which is not a part of the RAID. This is so I can dedicate the RAID to the MySQL server. The logical RAID device is mounted on the system as an ext4 partition without atime. This gives a lot better disk speed for random access which is key to SQL server performance.
I have so far clocked the system at ~60MB/sec transferring the databases from one server to the other via a closed point to point gigabit network link. This of course measuring the three bottlenecks: Read speed, Write speed and network transfer speed. Of course, this speed test probably flats out at the network transfer speed. It’s my fair guess, I will do more benchmarking later on.

The performance I have gotten out of these servers compared to the old servers, running some heavy sql queries from the actual system they are built to house, I have gotten results of anything from double performance to 8 times the performance.  It seems almost like the performance ratio increases exponentially by the heaviness of the sql query.

I also tried running some background batch processing on the servers. A process which normally gives the current servers ~7.5 load (on two cores) gives these servers 0.5 load (on four cores) and performs the task at 2.5 times the speed. The process is not threaded, except for the database. The processing itself is single thread and it is what loads the processor.

I will be updating the story of these two servers as I go along. They are estimated to leave our office tomorrow or the day after tomorrow to be put into a new co-location. They should be running the system in question before the end of November.

Story continued here