OceanBase 加入小组
【故障处理】安装OB集群，进行到步骤bootstrap ob时报错
【故障描述】
创建OB集群时，进行到 bootstrap ob 步骤时，出现错误，截图如下：
相关报错如下：
2021-03-22 20:25:32.120 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=root, cmd=pgrep observer | xargs --no-run-if-empty kill -9
2021-03-22 20:25:32.294 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=
2021-03-22 20:25:32.320 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] com.alipay.ocp.core.util.TaskUtils : [TaskUtils] wait 10 seconds
2021-03-22 20:25:42.389 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/*/*
2021-03-22 20:25:43.177 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=
2021-03-22 20:25:43.314 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/clog_shm /home/admin/oceanbase/store/sandata/ilog_shm
2021-03-22 20:25:43.385 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=
2021-03-22 20:25:43.439 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=root, cmd=pgrep observer | xargs --no-run-if-empty kill -9
2021-03-22 20:25:43.585 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=
2021-03-22 20:25:43.605 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] com.alipay.ocp.core.util.TaskUtils : [TaskUtils] wait 10 seconds
2021-03-22 20:25:53.654 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/*/*
2021-03-22 20:25:54.564 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=
2021-03-22 20:25:54.673 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/clog_shm /home/admin/oceanbase/store/sandata/ilog_shm
2021-03-22 20:25:54.823 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=
2021-03-22 20:25:54.929 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=root, cmd=pgrep observer | xargs --no-run-if-empty kill -9
2021-03-22 20:25:55.155 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=
2021-03-22 20:25:55.180 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] com.alipay.ocp.core.util.TaskUtils : [TaskUtils] wait 10 seconds
2021-03-22 20:26:05.232 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/*/*
2021-03-22 20:26:05.304 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=
2021-03-22 20:26:05.373 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/clog_shm /home/admin/oceanbase/store/sandata/ilog_shm
2021-03-22 20:26:05.424 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=
2021-03-22 20:26:05.451 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.c.metadb.task.TaskInstanceEntity : task 1833 run with context Context(parallelIdx=-1, stringMap={cluster_version=2.2.75, cluster_name=sandata, startup_option_string=cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648, ob_version=2.2.75, service_name=sandata:1, ob_connect_address=192.168.2.71, task_operation=retry, zone3.idc=beijing_3, cluster_type=PRIMARY, ob_cluster_id=1, link_obproxy=false, zone1.idc=beijing_1, primary_zone=zone1;zone2;zone3, sql_port=2881, service_type=OB_CLUSTER, cluster_id=2, rpm_name=oceanbase-2.2.75-20210108170301.el7.x86_64.rpm, target_cluster_status=RUNNING, zone2.idc=beijing_2}, listMap={zone3.host_ids=[7], root_servers=[5, 6, 7], root_server_ips=[192.168.2.71, 192.168.2.72, 192.168.2.73], zone2.host_ids=[6], host_ids=[5, 6, 7], zone1.host_ids=[5], zone_names=[zone1, zone2, zone3]})
2021-03-22 20:26:05.462 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.c.metadb.task.TaskInstanceEntity : executor node ip:192.168.2.70
2021-03-22 20:26:05.623 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=root, cmd=ifconfig 2> /dev/null | grep '192.168.2.71' -B 1 | grep -E 'flags|HWaddr' | awk -F '[: ]+' '{print $1}'
2021-03-22 20:26:05.744 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=ens192
2021-03-22 20:26:05.786 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=admin, cmd=cd /home/admin/oceanbase; ulimit -s 10240; ulimit -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD='' /home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone1 -d /home/admin/oceanbase/store/sandata -l info -o'obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648'
2021-03-22 20:26:07.484 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=/home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone1 -d /home/admin/oceanbase/store/sandata -l info -oobconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648
devname: ens192
mysql port: 2881
rpc port: 2882
appname: sandata
zone: zone1
data_dir: /home/admin/oceanbase/store/sandata
log level: info
optstr: obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648
2021-03-22 20:26:07.584 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=root, cmd=ifconfig 2> /dev/null | grep '192.168.2.72' -B 1 | grep -E 'flags|HWaddr' | awk -F '[: ]+' '{print $1}'
2021-03-22 20:26:07.622 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=ens192
2021-03-22 20:26:07.649 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=admin, cmd=cd /home/admin/oceanbase; ulimit -s 10240; ulimit -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD='' /home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone2 -d /home/admin/oceanbase/store/sandata -l info -o'obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648'
2021-03-22 20:26:07.847 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=/home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone2 -d /home/admin/oceanbase/store/sandata -l info -oobconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648
devname: ens192
mysql port: 2881
rpc port: 2882
appname: sandata
zone: zone2
data_dir: /home/admin/oceanbase/store/sandata
log level: info
optstr: obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648
2021-03-22 20:26:07.933 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=root, cmd=ifconfig 2> /dev/null | grep '192.168.2.73' -B 1 | grep -E 'flags|HWaddr' | awk -F '[: ]+' '{print $1}'
2021-03-22 20:26:07.984 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=ens192
2021-03-22 20:26:08.027 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=admin, cmd=cd /home/admin/oceanbase; ulimit -s 10240; ulimit -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD='' /home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone3 -d /home/admin/oceanbase/store/sandata -l info -o'obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648'
2021-03-22 20:26:08.378 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=/home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone3 -d /home/admin/oceanbase/store/sandata -l info -oobconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648
devname: ens192
mysql port: 2881
rpc port: 2882
appname: sandata
zone: zone3
data_dir: /home/admin/oceanbase/store/sandata
log level: info
optstr: obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648
2021-03-22 20:26:08.408 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.s.t.b.c.helper.ObServerTaskHelper : begin to check server accessible, hostIds=[5, 6, 7]
2021-03-22 20:26:08.456 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot -e "select 1 from dual;"
2021-03-22 20:26:08.789 ERROR 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : call pos method failed, server IP: 192.168.2.71, port: 62888, args: admin mysql -h127.1 -P2881 -uroot -e "select 1 from dual;", code: 1, result: ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111)
2021-03-22 20:26:08.819 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot -e "select 1 from dual;"
2021-03-22 20:26:08.928 ERROR 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : call pos method failed, server IP: 192.168.2.72, port: 62888, args: admin mysql -h127.1 -P2881 -uroot -e "select 1 from dual;", code: 1, result: ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111)
2021-03-22 20:26:08.972 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot -e "select 1 from dual;"
2021-03-22 20:26:09.211 ERROR 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : call pos method failed, server IP: 192.168.2.73, port: 62888, args: admin mysql -h127.1 -P2881 -uroot -e "select 1 from dual;", code: 1, result: ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111)
2021-03-22 20:26:09.225 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.s.t.b.c.helper.ObServerTaskHelper : there exists server(s) still not accessible, hostIds=5,6,7
2021-03-22 20:26:09.262 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] com.alipay.ocp.common.pattern.Retry : wait for 30 seconds
2021-03-22 20:26:39.343 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot -e "select 1 from dual;"
2021-03-22 20:26:40.734 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=1
1
2021-03-22 20:26:40.778 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot -e "select 1 from dual;"
2021-03-22 20:26:42.477 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=1
1
2021-03-22 20:26:42.555 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot -e "select 1 from dual;"
2021-03-22 20:26:44.032 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] result=1
1
2021-03-22 20:26:44.045 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.s.t.b.c.helper.ObServerTaskHelper : all ob server accessible
2021-03-22 20:26:44.055 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.s.t.b.cluster.BootStrapObTask : bootstrap command: mysql -h127.1 -P2881 -uroot -e "set ob_query_timeout=600000000; ALTER SYSTEM BOOTSTRAP REGION 'beijing' ZONE 'zone1' SERVER '192.168.2.71:2882', REGION 'beijing' ZONE 'zone2' SERVER '192.168.2.72:2882', REGION 'beijing' ZONE 'zone3' SERVER '192.168.2.73:2882' ;"
2021-03-22 20:26:44.088 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot -e "set ob_query_timeout=600000000; ALTER SYSTEM BOOTSTRAP REGION 'beijing' ZONE 'zone1' SERVER '192.168.2.71:2882', REGION 'beijing' ZONE 'zone2' SERVER '192.168.2.72:2882', REGION 'beijing' ZONE 'zone3' SERVER '192.168.2.73:2882' ;"
2021-03-22 20:36:44.603 ERROR 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils : call pos method failed, server IP: 192.168.2.71, port: 62888, args: admin mysql -h127.1 -P2881 -uroot -e "set ob_query_timeout=600000000; ALTER SYSTEM BOOTSTRAP REGION 'beijing' ZONE 'zone1' SERVER '192.168.2.71:2882', REGION 'beijing' ZONE 'zone2' SERVER '192.168.2.72:2882', REGION 'beijing' ZONE 'zone3' SERVER '192.168.2.73:2882' ;", code: 1, result: ERROR 4012 (HY000) at line 1: Timeout
2021-03-22 20:36:44.648 ERROR 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.core.task.manager.TaskManagerImpl : got exception while execute task:
com.alipay.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=COMMON_RPC_EXECUTE_FAILED, args=192.168.2.71,62888,admin mysql -h127.1 -P2881 -uroot -e "set ob_query_timeout=600000000; ALTER SYSTEM BOOTSTRAP REGION 'beijing' ZONE 'zone1' SERVER '192.168.2.71:2882', REGION 'beijing' ZONE 'zone2' SERVER '192.168.2.72:2882', REGION 'beijing' ZONE 'zone3' SERVER '192.168.2.73:2882' ;",1,ERROR 4012 (HY000) at line 1: Timeout
at com.alipay.ocp.core.task.util.OcpAgentUtils.executePosMethod(OcpAgentUtils.java:265) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
at com.alipay.ocp.core.task.util.OcpAgentUtils.callPosMethod(OcpAgentUtils.java:128) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
at com.alipay.ocp.core.task.util.OcpAgentUtils.runCmd(OcpAgentUtils.java:50) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
at com.alipay.ocp.service.compute.host.HostManager.executeRpcCommand(HostManager.java:82) ~[ocp-service-2.5.1-20210220.jar!/:2.5.1-20210220]
at com.alipay.ocp.service.task.business.cluster.BootStrapObTask.run(BootStrapObTask.java:112) ~[ocp-service-2.5.1-20210220.jar!/:2.5.1-20210220]
at com.alipay.ocp.core.metadb.task.TaskInstanceEntity.run(TaskInstanceEntity.java:192) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
at com.alipay.ocp.core.task.manager.TaskManagerImpl.lambda$null$1(TaskManagerImpl.java:186) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
at com.alipay.ocp.core.task.manager.TaskManagerImpl.redirectOutputIfNotSysSchedule(TaskManagerImpl.java:241) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
at com.alipay.ocp.core.task.manager.TaskManagerImpl.lambda$executeTask$2(TaskManagerImpl.java:159) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_275]
at java.util.concurrent.FutureTask.run(F
utureTask.java:266) ~[na:1.8.0_275]
at com.alipay.ocp.core.thread.TraceDecorator.lambda$decorate$0(TraceDecorator.java:28) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_275]
at java.lang.Thread.run(Thread.java:874) ~[na:1.8.0_275]
2021-03-22 20:36:44.684 INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.c.metadb.task.TaskInstanceEntity : set state failed for task: 1833
【问题分析】
我们可以在observer上面测试一下相关命令是否可以正常执行：
[root@observer1 clonescripts]# mysql -h127.1 -P2881 -uroot -e "select 1 from dual"
+---+
| 1 |
+---+
| 1 |
+---+
[root@observer1 clonescripts]# mysql -h127.1 -P2881 -uroot
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 3221315916
Server version: 5.7.25 OceanBase 2.2.75 (r20210108170301-c8c037dd6352fa0d00bd8a7105a1b566a5bc790d) (Built Jan 8 2021 17:31:33)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> ALTER SYSTEM BOOTSTRAP REGION 'beijing' ZONE 'zone1' SERVER '192.168.2.71:2882', REGION 'beijing' ZONE 'zone2' SERVER '192.168.2.72:2882', REGION 'beijing' ZONE 'zone3' SERVER '192.168.2.73:2882' ;
ERROR 4015 (HY000): System error
MySQL [(none)]>
如果出现上述错误，我采用的方法是删除集群，删除obsever主机，重新配置来解决。
