1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249
|
################################################################################
# Validate that when the sender to which the receiver is connected loses
# majority, the receiver will switch to another healthy sender. Also validate
# that the sender which lost majority still has its connection details in
# replication_asynchronous_connection_failover table.
#
# Test:
# 0. The test requires 4 servers:
# server1: group primary
# server2: group secondary
# server3: single server
# server4: replica server
# 1. Deploy a 2 members group in single primary mode.
# 2. Configure replica to manage the asynchronous connection
# to group and server3.
# 3. Create and start the managed channel connected to server1.
# 4. Kill server1, the group will block due no majority.
# 5. Despite server2 has the second biggest weight, since
# the group has no majority, the replica will connect
# to server3.
# 6. Wait 30 seconds and validate that replica is still connected
# to server3.
# 7. Assert that group information did not change, since it
# lost the majority.
# 8. Restore the majority on the group by forcing its membership
# to only server2.
# 9. Since server2 has bigger weight than server3, the managed
# channel must reconnect to server2 now that the group has
# majority.
# 10. Clean up.
################################################################################
--source include/big_test.inc
--source include/not_valgrind.inc
--source include/have_group_replication_plugin.inc
--let $rpl_skip_group_replication_start= 1
--let $rpl_server_count= 4
--let $group_replication_group_name= aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
--source include/group_replication.inc
--echo
--echo ############################################################
--echo # 1. Deploy a 2 members group in single primary mode.
--let $rpl_connection_name= server1
--source include/rpl_connection.inc
--let $server1_uuid= `SELECT @@server_uuid`
--source include/start_and_bootstrap_group_replication.inc
--let $rpl_connection_name= server2
--source include/rpl_connection.inc
--let $server2_uuid= `SELECT @@server_uuid`
--source include/start_group_replication.inc
--echo
--echo ############################################################
--echo # 2. Configure replica to manage the asynchronous connection
--echo # to group and server3.
--let $rpl_connection_name= server4
--source include/rpl_connection.inc
--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 $group_replication_group_name GROUP_NAME
--eval SELECT asynchronous_connection_failover_add_managed('ch1', 'GroupReplication', '$group_replication_group_name', '127.0.0.1', $SERVER_MYPORT_1, '', 90, 70);
--replace_result $SERVER_MYPORT_3 SERVER_MYPORT_3
--eval SELECT asynchronous_connection_failover_add_source('ch1', '127.0.0.1', $SERVER_MYPORT_3, '', 50);
--let $assert_text= 'There is one row in performance_schema.replication_asynchronous_connection_failover for server1'
--let $assert_cond= [SELECT COUNT(*) count FROM performance_schema.replication_asynchronous_connection_failover WHERE port=$SERVER_MYPORT_1, count, 1] = 1
--source include/assert.inc
--let $assert_text= 'There is one row in performance_schema.replication_asynchronous_connection_failover for server3'
--let $assert_cond= [SELECT COUNT(*) count FROM performance_schema.replication_asynchronous_connection_failover WHERE port=$SERVER_MYPORT_3, count, 1] = 1
--source include/assert.inc
--let $assert_text= 'There is one row in performance_schema.replication_asynchronous_connection_failover_managed for server1'
--let $assert_cond= [SELECT COUNT(*) count FROM performance_schema.replication_asynchronous_connection_failover_managed WHERE managed_name="$group_replication_group_name", count, 1] = 1
--source include/assert.inc
--echo
--echo ############################################################
--echo # 3. Create and start the managed channel connected to server1.
--let $rpl_connection_name= server4
--source include/rpl_connection.inc
--replace_result $SERVER_MYPORT_1 SERVER_1_PORT
--eval CHANGE REPLICATION SOURCE TO SOURCE_HOST='127.0.0.1', SOURCE_USER='root', SOURCE_AUTO_POSITION=1, SOURCE_CONNECTION_AUTO_FAILOVER=1, SOURCE_PORT=$SERVER_MYPORT_1, SOURCE_RETRY_COUNT=0 FOR CHANNEL 'ch1'
--let $rpl_channel_name='ch1'
--source include/start_slave.inc
--let $rpl_channel_name=
--let $assert_text= Verify channel ch1 IO_THREAD is ON and connected to server1
--let $assert_cond= "[SELECT SERVICE_STATE FROM performance_schema.replication_connection_status WHERE channel_name=\'ch1\' AND source_uuid=\'$server1_uuid\', SERVICE_STATE, 1]" = "ON"
--source include/assert.inc
--let $wait_condition= SELECT COUNT(*)=3 FROM performance_schema.replication_asynchronous_connection_failover
--source include/wait_condition.inc
--echo
--echo ############################################################
--echo # 4. Kill server1, the group will block due no majority.
--let $rpl_connection_name= server1
--source include/rpl_connection.inc
--source include/kill_and_restart_mysqld.inc
--let $rpl_server_number= 1
--source include/rpl_reconnect.inc
--let $rpl_connection_name= server2
--source include/rpl_connection.inc
--let $group_replication_member_state= UNREACHABLE
--let $group_replication_member_id= $server1_uuid
--source include/gr_wait_for_member_state.inc
--echo
--echo ############################################################
--echo # 5. Despite server2 has the second biggest weight, since
--echo # the group has no majority, the replica will connect
--echo # to server3.
--let $rpl_connection_name= server3
--source include/rpl_connection.inc
--let $server3_uuid= `SELECT @@server_uuid`
--let $rpl_connection_name= server4
--source include/rpl_connection.inc
--echo Wait until ch1 IO_THREAD is connected to server3
--let $wait_condition= SELECT service_state="ON" FROM performance_schema.replication_connection_status WHERE channel_name="ch1" AND source_uuid="$server3_uuid"
--source include/wait_condition.inc
--let $wait_condition= SELECT COUNT(*)>=1 FROM performance_schema.error_log WHERE (error_code='MY-013682' OR error_code='MY-013683') AND data LIKE 'The % thread detected that the source % does not belong to the group majority, thence the channel % will try to connect to another source.'
--source include/wait_condition.inc
--echo
--echo ############################################################
--echo # 6. Wait 30 seconds and validate that replica is still connected
--echo # to server3.
--let $rpl_connection_name= server4
--source include/rpl_connection.inc
--sleep 30
--let $assert_text= Verify channel ch1 IO_THREAD is ON and connected to server3
--let $assert_cond= "[SELECT SERVICE_STATE FROM performance_schema.replication_connection_status WHERE channel_name=\'ch1\' AND source_uuid=\'$server3_uuid\', SERVICE_STATE, 1]" = "ON"
--source include/assert.inc
--echo
--echo ############################################################
--echo # 7. Assert that group information did not change, since it
--echo # lost the majority.
--let $rpl_connection_name= server4
--source include/rpl_connection.inc
--let $assert_text= 'There is one row in performance_schema.replication_asynchronous_connection_failover for server1'
--let $assert_cond= [SELECT COUNT(*) count FROM performance_schema.replication_asynchronous_connection_failover WHERE port=$SERVER_MYPORT_1 AND weight=90 AND managed_name="$group_replication_group_name" , count, 1] = 1
--source include/assert.inc
--let $assert_text= 'There is one row in performance_schema.replication_asynchronous_connection_failover for server2'
--let $assert_cond= [SELECT COUNT(*) count FROM performance_schema.replication_asynchronous_connection_failover WHERE port=$SERVER_MYPORT_2 AND weight=70 AND managed_name="$group_replication_group_name" , count, 1] = 1
--source include/assert.inc
--echo
--echo ############################################################
--echo # 8. Restore the majority on the group by forcing its membership
--echo # to only server2.
--let $rpl_connection_name= server2
--source include/rpl_connection.inc
--let $local_address_server2= `SELECT @@GLOBAL.group_replication_local_address`
--replace_result $local_address_server2 GROUP_REPLICATION_LOCAL_ADDR_SERVER2
--eval SET GLOBAL group_replication_force_members= "$local_address_server2"
# Check for the primary member after the group is unblocked
--let $server1_uuid= `SELECT @@server_uuid`
--let $group_replication_expected_uuid= $server2_uuid
--source include/gr_wait_primary_member_uuid.inc
--let $wait_condition=SELECT COUNT(*)=1 FROM performance_schema.replication_group_members where MEMBER_STATE="ONLINE"
--source include/wait_condition.inc
SET GLOBAL group_replication_force_members= "";
--echo
--echo ############################################################
--echo # 9. Since server2 has bigger weight than server3, the managed
--echo # channel must reconnect to server2 now that the group has
--echo # majority.
--let $rpl_connection_name= server4
--source include/rpl_connection.inc
--echo Wait until ch1 IO_THREAD is connected to server2
--let $wait_condition= SELECT service_state="ON" FROM performance_schema.replication_connection_status WHERE channel_name="ch1" AND source_uuid="$server2_uuid"
--source include/wait_condition.inc
--echo
--echo ############################################################
--echo # 10. Clean up.
--let $rpl_connection_name= server4
--source include/rpl_connection.inc
--let $rpl_channel_name='ch1'
--source include/stop_slave.inc
--let $rpl_reset_slave_all= 1
--let $rpl_multi_source= 1
--source include/rpl_reset_slave.inc
--let $rpl_channel_name=
--let $rpl_reset_slave_all=
--let $rpl_multi_source=
--replace_result $group_replication_group_name GROUP_NAME
--eval SELECT asynchronous_connection_failover_delete_managed('ch1', '$group_replication_group_name')
--let $assert_text= 'There are 0 rows in performance_schema.replication_asynchronous_connection_failover_managed for group'
--let $assert_cond= [SELECT COUNT(*) count FROM performance_schema.replication_asynchronous_connection_failover_managed WHERE managed_name="$group_replication_group_name", count, 1] = 0
--source include/assert.inc
--let $assert_text= 'There are 0 rows in performance_schema.replication_asynchronous_connection_failover for group'
--let $assert_cond= [SELECT COUNT(*) count FROM performance_schema.replication_asynchronous_connection_failover WHERE managed_name="$group_replication_group_name" , count, 1] = 0
--replace_result $SERVER_MYPORT_3 SERVER_MYPORT_3
--eval SELECT asynchronous_connection_failover_delete_source('ch1', '127.0.0.1', $SERVER_MYPORT_3, '');
--let $assert_text= 'There are no rows in performance_schema.replication_asynchronous_connection_failover'
--let $assert_cond= [SELECT COUNT(*) count FROM performance_schema.replication_asynchronous_connection_failover, count, 1] = 0
--source include/assert.inc
--let $assert_text= 'There are no rows in performance_schema.replication_asynchronous_connection_failover_managed'
--let $assert_cond= [SELECT COUNT(*) count FROM performance_schema.replication_asynchronous_connection_failover_managed, count, 1] = 0
--source include/assert.inc
SET SESSION sql_log_bin = 0;
call mtr.add_suppression("The source .* for channel 'ch1' has joined the group .*, and so added its entry into replication_asynchronous_connection_failover table.");
call mtr.add_suppression("The Monitor IO thread detected that the source .* does not belong to the group majority, thence the channel .* will try to connect to another source.*");
call mtr.add_suppression("The IO thread detected that the source .* does not belong to the group majority, thence the channel .* will try to connect to another source.");
call mtr.add_suppression("The source .* for channel 'ch1' has left the group .*, and so removed its entry from replication_asynchronous_connection_failover table.");
call mtr.add_suppression("The group .* for the channel .* has been removed, and so removed its entry from replication_asynchronous_connection_failover_managed and all the group members from replication_asynchronous_connection_failover table.");
call mtr.add_suppression("The group .* for the channel .* has been added, and so added its entry in replication_asynchronous_connection_failover_managed and source to replication_asynchronous_connection_failover table.");
call mtr.add_suppression("The Monitor IO thread failed to detect if the source belongs to the group majority on the source.*");
SET SESSION sql_log_bin = 1;
--let $rpl_connection_name= server2
--source include/rpl_connection.inc
--source include/stop_group_replication.inc
--source include/group_replication_end.inc
|