MYSQL主从复制部署MHA自动故障转移

一.环境描述

让客户满意是我们工作的目标，不断超越客户的期望值来自于我们对这个行业的热爱。我们立志把好的技术通过有效、简单的方式提供给客户，将通过不懈努力成为客户在信息化领域值得信任、有价值的长期合作伙伴，公司提供的服务项目有：国际域名空间、虚拟主机、营销软件、网站建设、镇安网站维护、网站推广。

操作系统：RHEL 6.4

MySQL   ：5.6.27

二、部署MHA

1.主库安装 MHA NODE

1.1安装需要的PERL模块

yum -y install *zlib-dev*

 install.sh

#!/bin/bash

wget http://xrl.us/cpanm --no-check-certificate

mv cpanm /usr/bin

chmod 755 /usr/bin/cpanm

cat >/root/list<

install DBD::mysql

EOF

for package in `cat /root/list`

do

cpanm $package

done

1.2安装 MHA NODE

从 https://github.com/yoshinorim/mha4mysql-node下载最新版本的 mha4mysql-node源代码（目前是 0.57版）

unzip mha4mysql-node-master.zip

cd mha4mysql-node-master

perl  Makefile.PL

make

make install

2.备库安装 MHA Manager

2.1安装需要的perl模块

yum -y install *zlib-dev*

 install.sh

#!/bin/bash

wget http://xrl.us/cpanm --no-check-certificate

mv cpanm /usr/bin

chmod 755 /usr/bin/cpanm

cat >/root/list<

install DBD::mysql

install Config::Tiny

install Log::Dispatch

install Parallel::ForkManager

install Time::HiRes

EOF

for package in `cat /root/list`

do

cpanm $package

done

2.2安装MHA NODE

从 https://github.com/yoshinorim/mha4mysql-node下载最新版本的 mha4mysql-node源代码（目前是 0.57版）

unzip mha4mysql-node-master.zip

cd mha4mysql-node-master

perl  Makefile.PL

make

make install

2.3安装 MHA Manager

从 https://github.com/yoshinorim/mha4mysql-manager下载最新版本的 mha4mysql-manager源代码（目前是 0.57版）

unzip mha4mysql-manager-master.zip

cd  mha4mysql-manager-master

perl  Makefile.PL

make

make install

3.配置 SSH登录无密码验证

3.1主库配置到 MHA manager节点（备库节点）SSH登录无密码验证

vi /etc/hosts

192.168.222.157  mysqldb2

192.168.222.156  oradb

ssh-keygen  -t rsa

ssh-coyp-id -i /root/.ssh/id_rsa.pub root@192.168.222.157

ssh-coyp-id -i /root/.ssh/id_rsa.pub root@192.168.222.156

ssh root@192.168.222.156

ssh root@oradb

ssh  root@192.168.222.157

ssh  root@mysqldb2

3.2 MHA manager节点（备库节点）配置到主库配置 SSH登录无密码验证

vi /etc/hosts

192.168.222.157  mysqldb2

192.168.222.156  oradb

ssh-keygen  -t rsa

ssh-coyp-id -i /root/.ssh/id_rsa.pub root@192.168.222.157

ssh-coyp-id -i /root/.ssh/id_rsa.pub root@192.168.222.156

ssh root@192.168.222.156

ssh root@oradb

ssh  root@192.168.222.157

ssh  root@mysqldb2

4.搭建主从复制

4.1 修改主库my.cnf配置服务器ID为10

vi /app/oracle/oradata2/mysql-5.6.27/my.cnf

[mysqld]

server_id = 10

4.1 修改备库my.cnf配置

vi /usr/local/mysql/my.cnf

[mysqld]

server_id = 10

# slave config

log_bin=/usr/local/mysql/data/mysql-bin

relay_log=/usr/local/mysql/data/relay-bin

binlog_format=row

relay_log_purge=0

read_only=1

4.2 在主库上备份

mysqldump  --master-data=2  --single-transaction  -R  -A > all.sql

其中 --master-data=2代表备份时刻记录 master的 Binlog位置和 Position，--single-transaction意思是获取一致性快照， -R意思是备份相关的存储过程

，-A代表备份所有的 schema。

4.3 在主库上创建复制用户和 MHA监控用户

mysql> grant replication slave on *.* to 'repl'@'192.168.222.%' identified by '123456';

mysql> grant  all on *.*  to  'root'@'192.168.222.%'   identified by '123456'; /*MHA监控用户*/

4.4 查看主库上备份时刻 Binlog的名称和位置，MASTER_LOG和 MASTER_LOG_POS

head -n 30 all.sql|grep -i "CHANGE MASTER TO"

-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000068', MASTER_LOG_POS=325;

4.5 将备份复制到备库

scp all.sql root@192.168.222.157:/root/olm

4.6 在备库导入数据

mysql> source /root/olm/all.sql

mysql> flush privileges;

mysql> 

change master to master_host='192.168.222.156',master_user='repl',master_password='123456',

master_port=3306,master_log_file='mysql-bin.000068',master_log_pos=325;

start  slave;

--查看复制状态

show slave status \G

5.配置MHA

5.1 创建 MHA工作目录，并且创建相关配置文件

mkdir  -p  /etc/masterha

vi  /etc/masterha/app1.cnf

[server default]

manager_log=/masterha/app1/app1.log

manager_workdir=/masterha/app1

master_binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data

master_ip_failover_script=/usr/local/bin/master_ip_failover

master_ip_online_change_script=/usr/local/bin/master_ip_online_change

password=123456

ping_interval=1

remote_workdir=/tmp

repl_password=123456

repl_user=repl

secondary_check_script=/usr/local/bin/masterha_secondary_check -s mysqldb2 -s oradb --user=root --master_host=oradb --master_ip='192.168.222.156' --master_port=3306

shutdown_script=""

ssh_user=root

user=root

[server1]

hostname=192.168.222.156

port=3306

[server2]

candidate_master=1

check_repl_delay=0

hostname=192.168.222.157

port=3306

5.2 设置定期清理中继日志的脚本

crontab -e

0 4 * * * /usr/local/bin/purge_relay_logs --user=root --password=123456 --disable_replay_log_purge  --port=3306 --workdir=/usr/local/mysql/data  --disable_relay_log_purge >> /usr/loca/masterha/log/purge_relay_logs.log 2>&1

5.2 编辑 /etc/bashrc文件，在文件末尾添加以下内容，避免MHA执行故障切换时无法找到 mysqlbinlog

PATH="$PATH:/usr/local/mysql/bin"

export PATH

5.3 检查 SSH的配置

masterha_check_ssh  --conf=/etc/masterha/app1.cnf

5.4 检查MYSQL复制

masterha_check_repl  --conf=/etc/masterha/app1.cnf

5.5 启动 MHA Manger监控

tail -f /masterha/app1/app1.log

--在另一个SSH窗口启动 MHA Manger监控。--remove_dead_master_conf该参数代表当发生主从切换后，老的主库的IP将会从配置文件中移除。

--ignore_last_failover 在缺省情况下，如果MHA检测到连续发生宕机，而且两次宕机时间不足8小时的话，则不会进行 Failover，之所以这样限制

--是为了避免 ping-pong效应。该参数代表忽悠上次 MHA触发切换产生的文件，默认情况下，MHA发生切换后将会在 /masterha/paa1下产生 app1.

--failover.complete文件，下次再次切换的时候如果发现目录存在该文件将不允许触发切换，除非在第一次切换后收到 rm -rf /masterha/paa1/app1. --failover.complete,出于方便考虑，我们每次启动MHA时会添加 --ignore_last_failover参数。

nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover< /dev/null >/masterha/app1/manager.log 2>&1 &

--查看 MHA Manager监控是否正常

masterha_check_status --conf=/etc/masterha/app1.cnf

app1 (pid:4758) is running(0:PING_OK), master:192.168.222.156

--在默认情况下，10秒内状态为10：INITIALIZING_MONITOR，当状态转变为0:PING_OK后表明已经开启了到 MASTER端的监控。

关闭MHA的命令如下：

masterha_stop  --conf=/etc/masterha/app1.cnf

6.配置 VIP

6.1 在主库下载集群心跳软件 keepalived，并进行安装

访问 http://www.keepalived.org/software/ 下载 keepalived-1.2.18.tar.gz

tar vzxf keepalived-1.2.18.tar.gz

cd keepalived-1.2.18

./configure --prefix=/usr/local/keepalived

make

make install

cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

cd /etc/keepalived/

vi  keepalived.conf

--下面是添加的配置

[root@oradb ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

  notification_email {

    acassen@firewall.loc

    failover@firewall.loc

    sysadmin@firewall.loc

  }

  notification_email_from Alexandre.Cassen@firewall.loc

  smtp_server 127.0.0.1

  smtp_connect_timeout 30

  router_id MySQL-ha

}

vrrp_instance VI_1 {

   state BACKUP

   interface eth0

   virtual_router_id 51

   priority 150

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 1111

   }

   virtual_ipaddress {

       192.168.222.158/24

   }

}

service keepalived start

--是用 ip add命令可以看到 eth0上已经绑定了虚拟IP 192.168.222.158，注意使用 ifconfig 是看不到虚拟IP的

[root@oradb ~]# ip add

1: lo: mtu 16436 qdisc noqueue state UNKNOWN

   link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

   inet 127.0.0.1/8 scope host lo

   inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever

2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000

   link/ether 00:0c:29:53:7e:f0 brd ff:ff:ff:ff:ff:ff

   inet 192.168.222.156/24 brd 192.168.222.255 scope global eth0

   inet 192.168.222.158/24 scope global secondary eth0

   inet6 fe80::20c:29ff:fe53:7ef0/64 scope link

      valid_lft forever preferred_lft forever

3: eth2: mtu 1500 qdisc pfifo_fast state UP qlen 1000

   link/ether 00:0c:29:53:7e:fa brd ff:ff:ff:ff:ff:ff

   inet 192.168.1.11/24 brd 192.168.1.255 scope global eth2

   inet6 fe80::20c:29ff:fe53:7efa/64 scope link

      valid_lft forever preferred_lft forever

6.2 在备库下载集群心跳软件 keepalived，并进行安装

访问 http://www.keepalived.org/software/ 下载 keepalived-1.2.18.tar.gz

tar vzxf keepalived-1.2.18.tar.gz

cd keepalived-1.2.18

./configure --prefix=/usr/local/keepalived

make

make install

cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

cd /etc/keepalived/

vi  keepalived.conf

--下面是添加的配置

! Configuration File for keepalived

global_defs {

  notification_email {

    acassen@firewall.loc

    failover@firewall.loc

    sysadmin@firewall.loc

  }

  notification_email_from Alexandre.Cassen@firewall.loc

  smtp_server 127.0.0.1

  smtp_connect_timeout 30

  router_id MySQL-ha

}

vrrp_instance VI_1 {

   state BACKUP

   interface eth0

   virtual_router_id 51

   priority 120

   advert_int 1

   authentication {

       auth_type PASS

       auth_pass 1111

   }

   virtual_ipaddress {

       192.168.222.158/24

   }

}

service keepalived start

6.2 MHA引入 keepalived

在MHA监控节点（备库）编辑 master_ip_failover脚本

vi /usr/local/bin/master_ip_failover

#!/usr/bin/env perl

# Copyright (C) 2011 DeNA Co.,Ltd.

#

# This program is free software; you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation; either version 2 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

#  along with this program; if not, write to the Free Software

# Foundation, Inc.,

# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

## Note: This is a sample script and is not complete. Modify the script based on your environment.

use strict;

use warnings FATAL => 'all';

use Getopt::Long;

use MHA::DBHelper;

my (

 $command,       $ssh_user,        $orig_master_host,

 $orig_master_ip, $orig_master_port, $new_master_host,

 $new_master_ip, $new_master_port, $new_master_user,

 $new_master_password, $cmd

);

GetOptions(

 'command=s'            => \$command,

 'ssh_user=s'           => \$ssh_user,

 'orig_master_host=s'   => \$orig_master_host,

 'orig_master_ip=s'     => \$orig_master_ip,

 'orig_master_port=i'   => \$orig_master_port,

 'new_master_host=s'    => \$new_master_host,

 'new_master_ip=s'      => \$new_master_ip,

 'new_master_port=i'    => \$new_master_port,

 'new_master_user=s'    => \$new_master_user,

 'new_master_password=s' => \$new_master_password,

);

exit &main();

sub main {

 if ( $command eq "stop" || $command eq "stopssh" ) {

   # $orig_master_host, $orig_master_ip, $orig_master_port are passed.

   # If you manage master ip address at global catalog database,

   # invalidate orig_master_ip here.

   my $exit_code = 1;

   eval {

     # updating global catalog, etc

     $exit_code = 0;

   };

   if ($@) {

     warn "Got Error: $@\n";

     exit $exit_code;

   }

   exit $exit_code;

 }

 elsif ( $command eq "start" ) {

   # all arguments are passed.

   # If you manage master ip address at global catalog database,

   # activate new_master_ip here.

   # You can also grant write access (create user, set read_only=0, etc) here.

   my $exit_code = 10;

   eval {

     my $new_master_handler = new MHA::DBHelper();

     # args: hostname, port, user, password, raise_error_or_not

     $new_master_handler->connect( $new_master_ip, $new_master_port,

      $new_master_user, $new_master_password, 1 );

     ## Set read_only=0 on the new master

     $new_master_handler->disable_log_bin_local();

     print "Set read_only=0 on the new master.\n";

     $new_master_handler->disable_read_only();

     ## Creating an app user on the new master

     print "Creating app user on the new master..\n";

     #FIXME_xxx_create_user( $new_master_handler->{dbh} );

     #$new_master_handler->enable_log_bin_local();

     #$new_master_handler->disconnect();

     ## Update master ip on the catalog database, etc

     #FIXME_xxx;

     #$cmd = 'ssh root@192.168.222.156 \./lvs-admin stop';

     #$cmd = 'ssh'.$ssh_user.'@'.$orig_master_ip.'\'./lvs-admin stop\'';

     $cmd = 'ssh '.$ssh_user.'@'.$orig_master_ip.'  \./lvs-admin stop';

     system($cmd);

     $exit_code = 0;

   };

   if ($@) {

     warn $@;

     # If you want to continue failover, exit 10.

     exit $exit_code;

   }

   exit $exit_code;

 }

 elsif ( $command eq "status" ) {

   # do nothing

   exit 0;

 }

 else {

   &usage();

   exit 1;

 }

}

sub usage {

 print

"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";

}

在脚本中绿色背景的代码是新增的，红色背景的代码是注释的。

在主库添加关闭 keepalived的脚本

vi /root/lvs-admin

case "$1" in

     "stop")

     echo;

     echo "stop keepalived ......";

     service keepalived stop

     ;;

     "start")

     echo;

     echo "start keepalive ......";

     service keepalived start

     ;;

     *)

     echo "please input you select!"

     ;;

esac

chmod 755 /root/lvs-admin

7.测试主库故障MHA自动把备库提升为主库并把虚拟IP漂移到新的主库（原来的备库）

7.1在备库停止复制默认有二进制日志没传到备库

mysql>  stop slave;

7.2 sysbench生成测试数据，在主库上的 sbtest库下创建 sbtest表，共100万条数据

mysql>  create database sbtest;

./sysbench --num-threads=4 --test=oltp --oltp-table-size=1000000 --oltp-read-only=off --init-rng=on --max-requests=0 \

--oltp-dist-type=uniform --max-time=1800 --mysql-user=root --db-driver=test --mysql-table-engine=innodb \

--oltp-test-mode=complex prepare

7.3 关闭主库 MHA会自动把备份库提升为新的主库，并把虚拟IP漂移到新的主库（原来的备库）

--关闭主库

mysqladmin -u root shutdown

--在MHA监控节点（备库）实时查看MHA日志

tail  -f /masterha/app1/app1.log

Thu Feb 2 23:53:06 2017 - [info] MHA::MasterMonitor version 0.57.

Thu Feb 2 23:53:07 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Thu Feb 2 23:53:07 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Thu Feb 2 23:53:07 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Thu Feb 2 23:53:07 2017 - [info] Got exit code 1 (Not master dead).

Thu Feb 2 23:54:15 2017 - [info] MHA::MasterMonitor version 0.57.

Thu Feb 2 23:54:16 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Thu Feb 2 23:54:16 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Thu Feb 2 23:54:16 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Thu Feb 2 23:54:16 2017 - [info] Got exit code 1 (Not master dead).

Thu Feb 2 23:58:13 2017 - [info] MHA::MasterMonitor version 0.57.

Thu Feb 2 23:58:14 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Thu Feb 2 23:58:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Thu Feb 2 23:58:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Thu Feb 2 23:58:14 2017 - [info] Got exit code 1 (Not master dead).

Thu Feb 2 23:59:48 2017 - [info] MHA::MasterMonitor version 0.57.

Thu Feb 2 23:59:49 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Thu Feb 2 23:59:49 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Thu Feb 2 23:59:49 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Thu Feb 2 23:59:49 2017 - [info] Got exit code 1 (Not master dead).

Fri Feb 3 00:04:10 2017 - [info] MHA::MasterMonitor version 0.57.

Fri Feb 3 00:04:11 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Fri Feb 3 00:04:11 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Fri Feb 3 00:04:11 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Fri Feb 3 00:04:11 2017 - [info] Got exit code 1 (Not master dead).

Fri Feb 3 00:09:04 2017 - [info] MHA::MasterMonitor version 0.57.

Fri Feb 3 00:09:05 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Fri Feb 3 00:09:05 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Fri Feb 3 00:09:05 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Fri Feb 3 00:09:05 2017 - [info] Got exit code 1 (Not master dead).

Fri Feb 3 00:13:13 2017 - [info] MHA::MasterMonitor version 0.57.

Fri Feb 3 00:13:14 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Fri Feb 3 00:13:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Fri Feb 3 00:13:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Fri Feb 3 00:13:14 2017 - [info] Got exit code 1 (Not master dead).

Fri Feb 3 00:20:21 2017 - [info] MHA::MasterMonitor version 0.57.

Fri Feb 3 00:20:23 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Fri Feb 3 00:20:23 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Fri Feb 3 00:20:23 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Fri Feb 3 00:20:23 2017 - [info] Got exit code 1 (Not master dead).

Fri Feb 3 00:21:54 2017 - [info] MHA::MasterMonitor version 0.57.

Fri Feb 3 00:21:55 2017 - [info] GTID failover mode = 0

Fri Feb 3 00:21:55 2017 - [info] Dead Servers:

Fri Feb 3 00:21:55 2017 - [info] Alive Servers:

Fri Feb 3 00:21:55 2017 - [info]  192.168.222.156(192.168.222.156:3306)

Fri Feb 3 00:21:55 2017 - [info]  192.168.222.157(192.168.222.157:3306)

Fri Feb 3 00:21:55 2017 - [info] Alive Slaves:

Fri Feb 3 00:21:55 2017 - [info]  192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Fri Feb 3 00:21:55 2017 - [info]    Replicating from 192.168.222.156(192.168.222.156:3306)

Fri Feb 3 00:21:55 2017 - [info]    Primary candidate for the new Master (candidate_master is set)

Fri Feb 3 00:21:55 2017 - [info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

Fri Feb 3 00:21:55 2017 - [info] Checking slave configurations..

Fri Feb 3 00:21:55 2017 - [info] Checking replication filtering settings..

Fri Feb 3 00:21:55 2017 - [info] binlog_do_db= , binlog_ignore_db=

Fri Feb 3 00:21:55 2017 - [info] Replication filtering check ok.

Fri Feb 3 00:21:55 2017 - [info] GTID (with auto-pos) is not supported

Fri Feb 3 00:21:55 2017 - [info] Starting SSH connection tests..

Fri Feb 3 00:21:58 2017 - [info] All SSH connection tests passed successfully.

Fri Feb 3 00:21:58 2017 - [info] Checking MHA Node version..

Fri Feb 3 00:21:58 2017 - [info] Version check ok.

Fri Feb 3 00:21:58 2017 - [info] Checking SSH publickey authentication settings on the current master..

Fri Feb 3 00:21:59 2017 - [info] HealthCheck: SSH to 192.168.222.156 is reachable.

Fri Feb 3 00:21:59 2017 - [info] Master MHA Node version is 0.57.

Fri Feb 3 00:21:59 2017 - [info] Checking recovery script configurations on 192.168.222.156(192.168.222.156:3306)..

Fri Feb 3 00:21:59 2017 - [info]  Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=

/tmp/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000068

Fri Feb 3 00:21:59 2017 - [info]  Connecting to root@192.168.222.156(192.168.222.156:22)..

 Creating /tmp if not exists..   ok.

 Checking output directory is accessible or not..

  ok.

 Binlog found at /app/oracle/oradata2/mysql-5.5.37/data, up to mysql-bin.000068

Fri Feb 3 00:21:59 2017 - [info] Binlog setting check done.

Fri Feb 3 00:21:59 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Fri Feb 3 00:21:59 2017 - [info]  Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.222.157 --slave_ip=192.168.222.15

7 --slave_port=3306 --workdir=/tmp --target_version=5.6.27-log --manager_version=0.57 --relay_log_info=/usr/local/mysql/data/relay-log.info --relay_dir=/usr/local/mys

ql/data/ --slave_pass=xxx

Fri Feb 3 00:21:59 2017 - [info]  Connecting to root@192.168.222.157(192.168.222.157:22)..

 Checking slave recovery environment settings..

   Opening /usr/local/mysql/data/relay-log.info ... ok.

   Relay log found at /usr/local/mysql/data, up to relay-bin.000002

   Temporary relay log file is /usr/local/mysql/data/relay-bin.000002

   Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

   Testing mysqlbinlog output.. done.

   Cleaning up test file(s).. done.

Fri Feb 3 00:22:00 2017 - [info] Slaves settings check done.

Fri Feb 3 00:22:00 2017 - [info]

192.168.222.156(192.168.222.156:3306) (current master)

 +--192.168.222.157(192.168.222.157:3306)

Fri Feb 3 00:22:00 2017 - [info] Checking master_ip_failover_script status:

Fri Feb 3 00:22:00 2017 - [info]  /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.

156 --orig_master_port=3306

Fri Feb 3 00:22:00 2017 - [info] OK.

Fri Feb 3 00:22:00 2017 - [warning] shutdown_script is not defined.

Fri Feb 3 00:22:00 2017 - [info] Set master ping interval 1 seconds.

Fri Feb 3 00:22:00 2017 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s mysqldb2 -s oradb --user=root --master_host=oradb --master_ip=

'192.168.222.156' --master_port=3306

Fri Feb 3 00:22:00 2017 - [info] Starting ping health check on 192.168.222.156(192.168.222.156:3306)..

Fri Feb 3 00:22:00 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

Fri Feb 3 00:25:06 2017 - [warning] Got error on MySQL select ping: 2013 (Lost connection to MySQL server during query)

Fri Feb 3 00:25:06 2017 - [info] Executing secondary network check script: /usr/local/bin/masterha_secondary_check -s mysqldb2 -s oradb --user=root --master_host=orad

b --master_ip='192.168.222.156' --master_port=3306 --user=root --master_host=192.168.222.156 --master_ip=192.168.222.156 --master_port=3306 --master_user=root --ma

ster_password=123456 --ping_type=SELECT

Fri Feb 3 00:25:06 2017 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --outpu

t_file=/tmp/save_binary_logs_test --manager_version=0.57 --binlog_prefix=mysql-bin

Fri Feb 3 00:25:07 2017 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

Fri Feb 3 00:25:07 2017 - [warning] Connection failed 2 time(s)..

Monitoring server mysqldb2 is reachable, Master is not reachable from mysqldb2. OK.

Fri Feb 3 00:25:08 2017 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

Fri Feb 3 00:25:08 2017 - [warning] Connection failed 3 time(s)..

Monitoring server oradb is reachable, Master is not reachable from oradb. OK.

Fri Feb 3 00:25:08 2017 - [info] Master is not reachable from all other monitoring servers. Failover should start.

Fri Feb 3 00:25:08 2017 - [info] HealthCheck: SSH to 192.168.222.156 is reachable.

Fri Feb 3 00:25:09 2017 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

Fri Feb 3 00:25:09 2017 - [warning] Connection failed 4 time(s)..

Fri Feb 3 00:25:09 2017 - [warning] Master is not reachable from health checker!

Fri Feb 3 00:25:09 2017 - [warning] Master 192.168.222.156(192.168.222.156:3306) is not reachable!

Fri Feb 3 00:25:09 2017 - [warning] SSH is reachable.

Fri Feb 3 00:25:09 2017 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and tryi

ng to connect to all servers to check server status..

Fri Feb 3 00:25:09 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Fri Feb 3 00:25:09 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Fri Feb 3 00:25:09 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Fri Feb 3 00:25:10 2017 - [warning] SQL Thread is stopped(no error) on 192.168.222.157(192.168.222.157:3306)

Fri Feb 3 00:25:10 2017 - [info] GTID failover mode = 0

Fri Feb 3 00:25:10 2017 - [info] Dead Servers:

Fri Feb 3 00:25:10 2017 - [info]  192.168.222.156(192.168.222.156:3306)

Fri Feb 3 00:25:10 2017 - [info] Alive Servers:

Fri Feb 3 00:25:10 2017 - [info]  192.168.222.157(192.168.222.157:3306)

Fri Feb 3 00:25:10 2017 - [info] Alive Slaves:

Fri Feb 3 00:25:10 2017 - [info]  192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Fri Feb 3 00:25:10 2017 - [info]    Replicating from 192.168.222.156(192.168.222.156:3306)

Fri Feb 3 00:25:10 2017 - [info]    Primary candidate for the new Master (candidate_master is set)

Fri Feb 3 00:25:10 2017 - [info] Checking slave configurations..

Fri Feb 3 00:25:10 2017 - [info] Checking replication filtering settings..

Fri Feb 3 00:25:10 2017 - [info] Replication filtering check ok.

Fri Feb 3 00:25:10 2017 - [info] Master is down!

Fri Feb 3 00:25:10 2017 - [info] Terminating monitoring script.

Fri Feb 3 00:25:10 2017 - [info] Got exit code 20 (Master dead).

Fri Feb 3 00:25:10 2017 - [info] MHA::MasterFailover version 0.57.

Fri Feb 3 00:25:10 2017 - [info] Starting master failover.

Fri Feb 3 00:25:10 2017 - [info]

Fri Feb 3 00:25:10 2017 - [info] * Phase 1: Configuration Check Phase..

Fri Feb 3 00:25:10 2017 - [info]

Fri Feb 3 00:25:11 2017 - [warning] SQL Thread is stopped(no error) on 192.168.222.157(192.168.222.157:3306)

Fri Feb 3 00:25:11 2017 - [info] GTID failover mode = 0

Fri Feb 3 00:25:11 2017 - [info] Dead Servers:

Fri Feb 3 00:25:11 2017 - [info]  192.168.222.156(192.168.222.156:3306)

Fri Feb 3 00:25:11 2017 - [info] Checking master reachability via MySQL(double check)...

Fri Feb 3 00:25:11 2017 - [info] ok.

Fri Feb 3 00:25:11 2017 - [info] Alive Servers:

Fri Feb 3 00:25:11 2017 - [info]  192.168.222.157(192.168.222.157:3306)

Fri Feb 3 00:25:11 2017 - [info] Alive Slaves:

Fri Feb 3 00:25:11 2017 - [info]  192.168.222.157(192.168.222.157:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Fri Feb 3 00:25:11 2017 - [info]    Replicating from 192.168.222.156(192.168.222.156:3306)

Fri Feb 3 00:25:11 2017 - [info]    Primary candidate for the new Master (candidate_master is set)

Fri Feb 3 00:25:11 2017 - [info] Starting SQL thread on 192.168.222.157(192.168.222.157:3306) ..

Fri Feb 3 00:25:11 2017 - [info]  done.

Fri Feb 3 00:25:11 2017 - [info] Starting Non-GTID based failover.

Fri Feb 3 00:25:11 2017 - [info]

Fri Feb 3 00:25:11 2017 - [info] ** Phase 1: Configuration Check Phase completed.

Fri Feb 3 00:25:11 2017 - [info]

Fri Feb 3 00:25:11 2017 - [info] * Phase 2: Dead Master Shutdown Phase..

Fri Feb 3 00:25:11 2017 - [info]

Fri Feb 3 00:25:11 2017 - [info] Forcing shutdown so that applications never connect to the current master..

Fri Feb 3 00:25:11 2017 - [info] Executing master IP deactivation script:

Fri Feb 3 00:25:11 2017 - [info]  /usr/local/bin/master_ip_failover --orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156 --orig_master_port=3306 --com

mand=stopssh --ssh_user=root 

Fri Feb 3 00:25:11 2017 - [info] done.

Fri Feb 3 00:25:11 2017 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.

Fri Feb 3 00:25:11 2017 - [info] * Phase 2: Dead Master Shutdown Phase completed.

Fri Feb 3 00:25:11 2017 - [info]

Fri Feb 3 00:25:11 2017 - [info] * Phase 3: Master Recovery Phase..

Fri Feb 3 00:25:11 2017 - [info]