Keepalived+HAProxy實現MySQL高可用負載均衡的配置

Keepalived

由于在生產環境使用了mysqlcluster,需要實現高可用負載均衡,這里提供了keepalived+haproxy來實現.

keepalived主要功能是實現真實機器的故障隔離及負載均衡器間的失敗切換.可在第3,4,5層交換.它通過VRRPv2(Virtual Router Redundancy Protocol) stack實現的.

Layer3:Keepalived會定期向服務器群中的服務器.發送一個ICMP的數據包（既我們平時用的Ping程序）,如果發現某臺服務的IP地址沒有激活，Keepalived便報告這臺服務器失效，并將它從服務器群中剔除，這種情況的典型例子是某臺服務器被非法關機。Layer3的方式是以服務器的IP地址是否有效作為服務器工作正常與否的標準。

Layer4:主要以TCP端口的狀態來決定服務器工作正常與否。如web server的服務端口一般是80，如果Keepalived檢測到80端口沒有啟動，則Keepalived將把這臺服務器從服務器群中剔除。

Layer5:在網絡上占用的帶寬也要大一些。Keepalived將根據用戶的設定檢查服務器程序的運行是否正常，如果與用戶的設定不相符，則Keepalived將把服務器從服務器群中剔除。

Software Design

Keepalived+HAProxy實現MySQL高可用負載均衡的配置

keepalived啟動后會有單個進程

8352 ?    Ss   0:00 /usr/sbin/keepalived8353 ?    S   0:00 /_ /usr/sbin/keepalived8356 ?    S   0:01 /_ /usr/sbin/keepalived

父進程:內存管理,子進程管理等等

子進程:VRRP子進程

子進程:Healthchecking 子進程
實例

2臺mysqlcluster 10.1.6.203 master 10.1.6.205 backup

vip 10.1.6.173

目的訪問10.1.6.173 3366端口分別輪詢通過haproxy轉發到10.1.6.203 3306 和10.1.6.205 3306

mysqlcluster搭建參照之前博客,這里在2臺機上安裝keepalived

root@10.1.6.203:~# apt-get install keepalivedroot@10.1.6.203:~# cat /etc/keepalived/keepalived.conf vrrp_script chk_haproxy {    script "killall -0 haproxy"  # verify the pid existance    interval 2          # check every 2 seconds    weight -2          # add 2 points of prio if OK} vrrp_instance VI_1 {    interface eth1        # interface to monitor    state MASTER             virtual_router_id 51     # Assign one ID for this route    priority 101         # 101 on master, 100 on backup    nopreempt    debug     virtual_ipaddress {        10.1.6.173    }     track_script {    #注意大括號空格        chk_haproxy    }     notify_master /etc/keepalived/scripts/start_haproxy.sh #表示當切換到master狀態時,要執行的腳本    notify_fault /etc/keepalived/scripts/stop_keepalived.sh #故障時執行的腳本    notify_stop  /etc/keepalived/scripts/stop_haproxy.sh #keepalived停止運行前運行notify_stop指定的腳本 }

VRRPD配置包括三個類:

VRRP同步組(synchroization group)
VRRP實例(VRRP Instance)
VRRP腳本

這里使用了 VRRP實例, VRRP腳本

注意配置選項:

stat:指定instance(Initial)的初始狀態，就是說在配置好后，這臺服務器的初始狀態就是這里指定的，但這里指定的不算，還是得要通過競選通過優先級來確定，里如果這里設置為master，但如若他的優先級不及另外一臺，那么這臺在發送通告時，會發送自己的優先級，另外一臺發現優先級不如自己的高，那么他會就回搶占為master

interface：實例綁定的網卡，因為在配置虛擬IP的時候必須是在已有的網卡上添加的
priority 101：設置本節點的優先級，優先級高的為master
debug：debug級別
nopreempt:設置為不搶占

vrrp_script chk_haproxy {    script "killall -0 haproxy"  # verify the pid existance    interval 2          # check every 2 seconds 腳本執行間隔    weight -2          # add 2 points of prio if OK 腳本結果導致的優先級變更：2表示優先級+2；-2則表示優先級-2}

然后在實例(vrrp_instance)里面引用，有點類似腳本里面的函數引用一樣：先定義，后引用函數名

    track_script {        chk_haproxy    }

注意：VRRP腳本(vrrp_script)和VRRP實例(vrrp_instance)屬于同一個級別

root@10.1.6.203:scripts# cat start_haproxy.sh #!/bin/bash sleep 5get=`ip addr |grep 10.1.6.173 |wc -l`echo $get >> /etc/keepalived/scripts/start_ha.log if [ $get -eq 1 ]then    echo "`date +%c` success to get vip" >> /etc/keepalived/scripts/start_ha.log    /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfgelse    echo "`date +%c` can not get vip" >> /etc/keepalived/scripts/start_ha.logfiroot@10.1.6.203:scripts# cat stop_keepalived.sh #!/bin/bash pid=`pidof keepalived`if [ $pid == "" ]then  echo "`date +%c` no keepalived process id" >> /etc/keepalived/scripts/stop_keep.logelse  echo "`date +%c` will stop keepalived " >> /etc/keepalived/scripts/stop_keep.log  /etc/init.d/keepalived stopfi /etc/init.d/keepalived stop root@10.1.6.203:scripts# cat stop_haproxy.sh #!/bin/bash pid=`pidof haproxy`echo "`date +%c` stop haproxy" >> /etc/keepalived/scripts/stop_ha.logkill -9 $pid

同理配置10.1.6.205

root@10.1.6.205:~# cat /etc/keepalived/keepalived.conf vrrp_script chk_haproxy {  script "killall -0 haproxy"  # verify the pid existance  interval 2          # check every 2 seconds  weight 2           # add 2 points of prio if OK} vrrp_instance VI_1 {  interface eth1        # interface to monitor  state BACKUP  virtual_router_id 51     # Assign one ID for this route  priority 100         # 101 on master, 100 on backup  virtual_ipaddress {    10.1.6.173  }   track_script {    chk_haproxy  } notify_master /etc/keepalived/scripts/start_haproxy.shnotify_fault /etc/keepalived/scripts/stop_keepalived.shnotify_stop /etc/keepalived/scripts/stop_haproxy.sh }

HAProxy

下面再介紹下haproxy

HAProxy是一款基于TCP(第四層)和HTTP(第七層)應用的代理軟件,它也可作為負載均衡器.可以支持數以萬計的并發連接.同時可以保護服務器不暴露到網絡上,通過端口映射.它還自帶監控服務器狀態的頁面.

安裝haproxy

wget -O/tmp/haproxy-1.4.22.tar.gz http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.22.tar.gztar xvfz /tmp/haproxy-1.4.22.tar.gz -C /tmp/cd /tmp/haproxy-1.4.22make TARGET=linux26make install

haproxy需要對每一個mysqlcluster服務器進行健康檢查

1.在2臺主機分別配置haproxy.cfg

root@10.1.6.203:scripts# cat /etc/haproxy/haproxy.cfg global    maxconn 51200 #默認最大連接數     #uid 99    #gid 99    daemon    #以后臺形式運行haproxy    #quiet    nbproc 1   #進程數量(可以設置多個進程提高性能)     pidfile /etc/haproxy/haproxy.pid #haproxy的pid存放路徑,啟動進程的用戶必須有權限訪問此文件  defaults    mode tcp      #所處理的類別 (#7層 http;4層tcp )     option redispatch  #serverId對應的服務器掛掉后,強制定向到其他健康的服務器     option abortonclose #當服務器負載很高的時候，自動結束掉當前隊列處理比較久的連接     timeout connect 5000s  #連接超時    timeout client 50000s #客戶端超時    timeout server 50000s  #服務器超時    log 127.0.0.1 local0  #錯誤日志記錄    balance roundrobin  #默認的負載均衡的方式,輪詢方式  listen proxy    bind 10.1.6.173:3366  #監聽端口     mode tcp        #http的7層模式    option httpchk    #心跳檢測的文件    server db1 10.1.6.203:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3   #服務器定義,check inter 12000是檢測心跳頻率 rise 3是3次正確認為服務器可用， fall 3是3次失敗認為服務器不可用，weight代表權重     server db2 10.1.6.205:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3 listen haproxy_stats    mode http    bind 10.1.6.173:8888    option httplog    stats refresh 5s      stats uri /status #網站健康檢測URL，用來檢測HAProxy管理的網站是否可以用，正常返回200，不正常返回503     stats realm Haproxy Manager    stats auth admin:p@a1SZs24 #賬號密碼root@10.1.6.205:~$ cat /etc/haproxy/haproxy.cfg global    maxconn 51200    #uid 99    #gid 99    daemon    #quiet    nbproc 1    pidfile /etc/haproxy/haproxy.pid defaults    mode tcp    option redispatch      option abortonclose    timeout connect 5000s    timeout client 50000s    timeout server 50000s    log 127.0.0.1 local0    balance roundrobin  listen proxy    bind 10.1.6.173:3366    mode tcp    option httpchk    server db1 10.1.6.203:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3    server db2 10.1.6.205:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3 listen haproxy_stats    mode http    bind 10.1.6.173:8888    option httplog    stats refresh 5s      stats uri /status     stats realm Haproxy Manager    stats auth admin:p@a1SZs24

2.安裝xinetd

root@10.1.6.203:~# apt-get install xinetd

3.在每個節點添加xinetd服務腳本和mysqlchk端口號

root@10.1.6.203:~# vim /etc/xinetd.d/mysqlchk # default: on# description: mysqlchkservice mysqlchk         #需要在servive定義{    flags      = REUSE    socket_type   = stream    port      = 9222    wait      = no    user      = nobody    server     = /opt/mysqlchk     log_on_failure += USERID    disable     = no    per_source   = UNLIMITED    bind      = 10.1.6.173} root@10.1.6.203:~# vim /etc/services mysqlchk    9222/tcp            # mysqlchk

4.編寫mysqlchk監控服務腳本

root@10.1.6.203:~# ls -l /opt/mysqlchk -rwxr--r-- 1 nobody root 1994 2013-09-17 11:27 /opt/mysqlchkroot@10.1.6.203:~# cat /opt/mysqlchk #!/bin/bash## This script checks if a mysql server is healthy running on localhost. It will# return:# "HTTP/1.x 200 OK/r" (if mysql is running smoothly)# - OR -# "HTTP/1.x 500 Internal Server Error/r" (else)## The purpose of this script is make haproxy capable of monitoring mysql properly# MYSQL_HOST="localhost"MYSQL_SOCKET="/var/run/mysqld/mysqld.sock"MYSQL_USERNAME="mysqlchkusr"   #該賬戶密碼需要在mysql里添加MYSQL_PASSWORD="secret"MYSQL_OPTS="-N -q -A"TMP_FILE="/dev/shm/mysqlchk.$$.out"ERR_FILE="/dev/shm/mysqlchk.$$.err"FORCE_FAIL="/dev/shm/proxyoff"MYSQL_BIN="/opt/mysqlcluster/mysql-cluster-gpl-7.2.6-linux2.6-x86_64/bin/mysql"CHECK_QUERY="select 1" preflight_check(){  for I in "$TMP_FILE" "$ERR_FILE"; do    if [ -f "$I" ]; then      if [ ! -w $I ]; then        echo -e "HTTP/1.1 503 Service Unavailable/r/n"        echo -e "Content-Type: Content-Type: text/plain/r/n"        echo -e "/r/n"        echo -e "Cannot write to $I/r/n"        echo -e "/r/n"        exit 1      fi    fi  done} return_ok(){  echo -e "HTTP/1.1 200 OK/r/n"  echo -e "Content-Type: text/html/r/n"  echo -e "Content-Length: 43/r/n"  echo -e "/r/n"  echo -e "<html><body>MySQL is running.</body></html>/r/n"  echo -e "/r/n"  rm $ERR_FILE $TMP_FILE  exit 0}return_fail(){  echo -e "HTTP/1.1 503 Service Unavailable/r/n"  echo -e "Content-Type: text/html/r/n"  echo -e "Content-Length: 42/r/n"  echo -e "/r/n"  echo -e "<html><body>MySQL is *down*.</body></html>/r/n"  sed -e 's//n$//r/n/' $ERR_FILE  echo -e "/r/n"  rm $ERR_FILE $TMP_FILE  exit 1}preflight_checkif [ -f "$FORCE_FAIL" ]; then    echo "$FORCE_FAIL found" > $ERR_FILE    return_fail;fi$MYSQL_BIN $MYSQL_OPTS --host=$MYSQL_HOST --socket=$MYSQL_SOCKET --user=$MYSQL_USERNAME --password=$MYSQL_PASSWORD -e "$CHECK_QUERY" > $TMP_FILE 2> $ERR_FILEif [ $? -ne 0 ]; then    return_fail;fireturn_ok;

測試

2個節點開啟keepalived(主節點會獲得vip,自動拉起haproxy),xinetd

root@10.1.6.203:~# ip add1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  inet 127.0.0.1/8 scope host lo2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000  link/ether 00:26:b9:36:0f:81 brd ff:ff:ff:ff:ff:ff  inet 211.151.105.186/26 brd 211.151.105.191 scope global eth03: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000  link/ether 00:26:b9:36:0f:83 brd ff:ff:ff:ff:ff:ff  inet 10.1.6.203/24 brd 10.1.6.255 scope global eth1  inet 10.1.6.173/32 scope global eth14: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000  link/ether 00:26:b9:36:0f:85 brd ff:ff:ff:ff:ff:ff5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000  link/ether 00:26:b9:36:0f:87 brd ff:ff:ff:ff:ff:ffroot@10.1.6.203:~# netstat -tunlp | grep hatcp    0   0 10.1.6.173:3366   0.0.0.0:*        LISTEN   1042/haproxy  tcp    0   0 10.1.6.203:8888   0.0.0.0:*        LISTEN   1042/haproxy  udp    0   0 0.0.0.0:56562      0.0.0.0:*              1042/haproxy  root@10.1.6.203:~# netstat -tunlp | grep xinetcp    0   0 10.1.6.203:9222   0.0.0.0:*        LISTEN   30897/xinetd  root@10.1.6.203:~# ps -ef | grep haproxyroot   1042   1 0 Sep17 ?    00:00:00 /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg

測試:

通過vip10.1.6.173 3366訪問cluster數據庫(注意賬戶dave權限需要加3個ip10.1.6.203,10.1.6.205,10.1.6.173)

root@10.1.6.203:mgm# mysql -udave -p -h 10.1.6.173 -P 3366Enter password: Welcome to the MySQL monitor. Commands end with ; or /g.Your MySQL connection id is 1344316Server version: 5.5.22-ndb-7.2.6-gpl-log MySQL Cluster Community Server (GPL) Type 'help;' or '/h' for help. Type '/c' to clear the buffer. mysql> show databases;+--------------------+| Database      |+--------------------+| information_schema | | dave       | | test        | +--------------------+3 rows in set (0.01 sec) mysql>

手動分別使keepalive,haproxy,數據庫掛掉.vip10.1.6.173會自動漂到10.1.6.205從上，并不影響vip的訪問

通過vip,haproxy查看各節點狀態

http://10.1.6.173:8888/status

Keepalived+HAProxy實現MySQL高可用負載均衡的配置