Jan 31 2005
Mysql Cluster: Two webserver setup (three servers required)
Notes
You MUST have a third server as a managment node but this can be shut down after the cluster starts. Also note that I do not recommend shutting down the managment server (see the extra notes at the bottom of this document for more information). You can not run a mysql cluster with just two servers.
I am going to talk about three servers,
mysql1.domain.com 192.168.0.1
mysql2.domain.com 192.168.0.2
mysql3.domain.com 192.168.0.3Servers 1 and 2 will be the two that end up “clustered”. This would be perfect for two servers behind a loadbalancer or using round robin DNS and is a good replacement for replication. Server 3 needs to have only minor changes made to it and does NOT require a mysql install. It can be a low-end machine and can be carrying out other tasks.
STAGE 1: Install mysql on the first two servers:
Complete the following steps on both mysql1 and mysql2:
cd /usr/local/
http://dev.mysql.com/get/Downloads/MySQL-4.1/mysql-max-4.1.9-pc-linux-gnu-i686.tar.gz/
from/http://www.signal42.com/mirrors/mysql/
groupadd mysql
useradd -g mysql mysql
tar -zxvf mysql-max-4.1.9-pc-linux-gnu-i686.tar.gz
rm mysql-max-4.1.9-pc-linux-gnu-i686.tar.gz
ln -s mysql-max-4.1.9-pc-linux-gnu-i686 mysql
cd mysql
scripts/mysql_install_db –user=mysql
chown -R root .
chown -R mysql data
chgrp -R mysql .
cp support-files/mysql.server /etc/rc.d/init.d/
chmod +x /etc/rc.d/init.d/mysql.server
chkconfig –add mysql.serverDo not start mysql yet.
STAGE 2: Install and configure the master server
You need the following files from the bin/ of the mysql directory: ndb_mgm and ndb_mgmd. Download the whole mysql-max tarball and extract them from the bin/ directory.
mkdir /usr/src/mysql-mgm
cd /usr/src/mysql-mgm
http://dev.mysql.com/get/Downloads/MySQL-4.1/mysql-max-4.1.9-pc-linux-gnu-i686.tar.gz/
from/http://www.signal42.com/mirrors/mysql/
tar -zxvf mysql-max-4.1.9-pc-linux-gnu-i686.tar.gz
rm mysql-max-4.1.9-pc-linux-gnu-i686.tar.gz
cd mysql-max-4.1.9-pc-linux-gnu-i686
mv bin/ndb_mgm .
mv bin/ndb_mgmd .
chmod +x ndb_mg*
mv ndb_mg* /usr/bin/
cd
rm -rf /usr/src/mysql-mgmYou now need to set up the config file for this master:
mkdir /var/lib/mysql-cluster
cd /var/lib/mysql-cluster
vi [or emacs or any other editor] config.iniNow, insert the following (changing the bits as indicated):
[NDBD DEFAULT]
NoOfReplicas=2
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
[TCP DEFAULT]
# Managment Server
[NDB_MGMD]
HostName=192.168.0.3 # the IP of THIS SERVER
# Storage Engines
[NDBD]
HostName=192.168.0.1 # the IP of the FIRST SERVER
DataDir= /var/lib/mysql-cluster
[NDBD]
HostName=192.168.0.2 # the IP of the SECOND SERVER
DataDir=/var/lib/mysql-cluster
# 2 MySQL Clients
# I personally leave this blank to allow rapid changes of the mysql clients;
# you can enter the hostnames of the above two servers here. I suggest you dont.
[MYSQLD]
[MYSQLD]Now, start the managment server:
ndb_mgmdThis is the mysql managment server, not maganment console. You should therefore not expect any output (we will start the console later).
STAGE 3: Configure the main two servers and start mysql
On each of the two main servers (192.168.0.1 and 192.168.0.2) enter the following (changing the bits as appropriate):
vi /etc/my.cnfEnter i to go to insert mode again and insert this on both servers (changing the IP address to the IP of the master server that you set up in stage 2):
[mysqld]
ndbcluster
ndb-connectstring=192.168.0.3 # the IP of the MANAGMENT (THIRD) SERVER
[mysql_cluster]
ndb-connectstring=192.168.0.3 # the IP of the MANAGMENT (THIRD) SERVERNow, we make the data directory and start the storage engine:
mkdir /var/lib/mysql-cluster
cd /var/lib/mysql-cluster
ndbd –initial
/etc/rc.d/init.d/mysql.server startIf you have done one server now go back to the start of stage 3 and repeat exactly the same procedure on the second server.
NOTE that you should ONLY use –initial if you are either starting from scratch or have changed the config.ini file on the master.
STAGE 4: Check its working
You can now return to the master server (mysql3) and enter the managment console:
ndb_mgmEnter the command SHOW to see what is going on. A sample output looks like this:
[root@mysql3 mysql-cluster]# ndb_mgm
– NDB Cluster — Management Client –
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=2 @192.168.0.1 (Version: 4.1.9, Nodegroup: 0, Master)
id=3 @192.168.0.2 (Version: 4.1.9, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @192.168.0.3 (Version: 4.1.9)
[mysqld(API)] 2 node(s)
id=4 (Version: 4.1.9)
id=5 (Version: 4.1.9)
ndb_mgm>If you see
not connected, accepting connect from 192.168.0.[1/2/3]in the first or last two lines they you have a problem. Please email me with as much detail as you can give and I can try to find out where you have gone wrong and change this HOWTO to fix it.
If you are OK to here it is time to test mysql. On either server mysql1 or mysql2 enter the following commands: Note that we have no root password yet.
mysql
use test;
CREATE TABLE ctest (i INT) ENGINE=NDBCLUSTER;
INSERT INTO ctest () VALUES (1);
SELECT * FROM ctest;You should see 1 row returned (with the value 1).
If this works, now go to the other server and run the same SELECT and see what you get. Insert from that host and go back to host 1 and see if it works. If it works then congratulations.
The final test is to kill one server to see what happens. If you have physical access to the machine simply unplug its network cable and see if the other server keeps on going fine (try the SELECT query). If you dont have physical access do the following:
ps aux | grep ndbdYou get an output like this:
root 5578 0.0 0.3 6220 1964 ? S 03:14 0:00 ndbd
root 5579 0.0 20.4 492072 102828 ? R 03:14 0:04 ndbd
root 23532 0.0 0.1 3680 684 pts/1 S 07:59 0:00 grep ndbdIn this case ignore the command “grep ndbd” (the last line) but kill the first two processes by issuing the command kill -9 pid pid:
kill -9 5578 5579Then try the select on the other server. While you are at it run a SHOW command on the master to see that the server has died. To restart it, just issue
ndbdNOTE no –inital!
——————————————————————————–
Further notes about setup
I strongly recommend that you read all of this (and bookmark this page). It will almost certainly save you a lot of searching.
The Managment Server
I strongly recommend that you do not stop the managment server once it has started. This is for several resons:
The server takes hardly any server resources
If a cluster falls over, you want to be able to just ssh in and type ndbd to stat it. You dont want to have to start messing around with another server
If you want to take backups then you need the managment server up
The cluster log is sent to the management server so to check what is going on in the cluster or has happened since last this is an important tool
All commands from the ndb_mgm client is sent to the management server and thus no management commands without management server.
The managment server is required in case of cluster reconfiguration (crashed server or network split). In the case that it is not running, “split-brain” scenario will occure. The management server arbitration role is required for this type of setup to provide better fault tollerance.
However you are welcome to stop the server if you prefer.
Starting and stopping ndbd automatically on boot
To achieve this, do the following on both mysql1 and mysql2:
echo “ndbd” > /etc/rc.d/init.d/ndbd
chmod +x /etc/rc.d/init.d/ndbd
chkconfig –add ndbdNote that this is a really quick script. You ought really to write one that at least checks if ndbd is already started on the machine.
Use of hostnames
You will note that I have used IP addresses exclusively throught this setup. This is because using hostnames simply increases the number of things that can go wrong. Mikael Ronström of MySQL AB kindly explains: “Hostnames certainly work with MySQL Cluster. But using hostnames introduces quite a few error sources since a proper DNS lookup system must be set-up, sometimes /etc/hosts must be edited and their might be security blocks ensuring that communication between certain machines is not possible other than on certain ports”. I strongly suggest that while testing you use IP addresses if you can, then once it is all working change to hostnames.
RAM
Use the following formula to work out the amount of RAM that you need on each storage node:
(Size of database * NumberofReplicas * 1.1) / Number of storage nodesNumberofReplicas is set to two by default. You can change it in config.ini if you want. So for example to run a 4GB database you need just under 9GB of RAM on each storage node. For the API nodes and managment nodes you dont need much RAM at all.
Adding API nodes
If you want to add another API node (i.e. you have another server that you want to add to the cluster but you dont need it to act as a storage node), then just add the following to /etc/my.cnf on the server (it must be a mysql-max server):
[mysqld]
ndbcluster
ndb-connectstring=192.168.0.3 # the IP of the MANAGMENT (THIRD) SERVER
[mysql_cluster]
ndb-connectstring=192.168.0.3 # the IP of the MANAGMENT (THIRD) SERVERThen you need to make sure that there is another [MYSQLD] line at the end of config.ini on the managment server. Restart the cluster (see below for an important note) and restart mysql on the new API. It should be connected.
Important note on changing config.ini
If you ever change config.ini you must stop the whole cluster and restart it with ndbd –initial to re-read the config file. Stop the cluster with a SHUTDOWN command to the ndb_mgm package on the managment server and then restart all the ndbd nodes.
Other resources
I found the following resources very useful:
The MySQL cluster documentation. This is gradually being reworked. I would, however, suggest that you at least read these pages:
On-line Backup of MySQL Cluster.
Defining MySQL Cluster Storage Nodes for information that you will need to allow for bigger database memory or a larger number of tables, indexes, unique indexes
MySQL Cluster mailing list.
Google.
MySQL Forums
The #mysql IRC chanel on freenode and EFNet. If you need a free (open source) IRC client I recomment Bersirc.
Thanks
I must thank several others who have contributed to this: Mikael Ronström from MySQL AB for helping me to get this to work and spotting my silly mistake right at the end, Lewis Bergman for proof-reading this page and pointing out some improvements, as well as suffering the frustration with me and Martin Pala for explaining the final reason to keep the managment server up as well as a few other minor changes. Thanks also to Terry from Advanced Network Hosts who paid me to set a cluster up and at the same time produce a HOWTO.
Comments/Corrections
I would be delighted for you to send any corrections or comments you may have to alex@davz.net. If you are also really stuck I would be delighted to help you.
Please do not copy this document; please just link to it. This is simply to prevent yet more old information getting out onto the internet. I will keep this updated. If you want something changed please email me.
Looking for exceptionally high quality, low cost and satisfaction guaranteed systems administration?
Contact Me for details.
See more HOWTOs like this