OceanBase 加入小组

15个成员 14个话题 创建时间:2021-02-09

【故障处理】安装OB集群,进行到步骤bootstrap ob时报错

发表于2021-03-22 2973次查看

【故障描述】

创建OB集群时,进行到 bootstrap ob 步骤时,出现错误,截图如下:

相关报错如下:

2021-03-22 20:25:32.120  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=root, cmd=pgrep observer | xargs --no-run-if-empty kill -9

2021-03-22 20:25:32.294  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=

2021-03-22 20:25:32.320  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] com.alipay.ocp.core.util.TaskUtils       : [TaskUtils] wait 10 seconds

2021-03-22 20:25:42.389  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/*/*

2021-03-22 20:25:43.177  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=

2021-03-22 20:25:43.314  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/clog_shm /home/admin/oceanbase/store/sandata/ilog_shm

2021-03-22 20:25:43.385  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=

2021-03-22 20:25:43.439  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=root, cmd=pgrep observer | xargs --no-run-if-empty kill -9

2021-03-22 20:25:43.585  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=

2021-03-22 20:25:43.605  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] com.alipay.ocp.core.util.TaskUtils       : [TaskUtils] wait 10 seconds

2021-03-22 20:25:53.654  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/*/*

2021-03-22 20:25:54.564  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=

2021-03-22 20:25:54.673  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/clog_shm /home/admin/oceanbase/store/sandata/ilog_shm

2021-03-22 20:25:54.823  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=

2021-03-22 20:25:54.929  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=root, cmd=pgrep observer | xargs --no-run-if-empty kill -9

2021-03-22 20:25:55.155  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=

2021-03-22 20:25:55.180  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] com.alipay.ocp.core.util.TaskUtils       : [TaskUtils] wait 10 seconds

2021-03-22 20:26:05.232  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/*/*

2021-03-22 20:26:05.304  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=

2021-03-22 20:26:05.373  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=root, cmd=rm -rf /home/admin/oceanbase/store/sandata/clog_shm /home/admin/oceanbase/store/sandata/ilog_shm

2021-03-22 20:26:05.424  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=

2021-03-22 20:26:05.451  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.c.metadb.task.TaskInstanceEntity   : task 1833 run with context Context(parallelIdx=-1, stringMap={cluster_version=2.2.75, cluster_name=sandata, startup_option_string=cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648, ob_version=2.2.75, service_name=sandata:1, ob_connect_address=192.168.2.71, task_operation=retry, zone3.idc=beijing_3, cluster_type=PRIMARY, ob_cluster_id=1, link_obproxy=false, zone1.idc=beijing_1, primary_zone=zone1;zone2;zone3, sql_port=2881, service_type=OB_CLUSTER, cluster_id=2, rpm_name=oceanbase-2.2.75-20210108170301.el7.x86_64.rpm, target_cluster_status=RUNNING, zone2.idc=beijing_2}, listMap={zone3.host_ids=[7], root_servers=[5, 6, 7], root_server_ips=[192.168.2.71, 192.168.2.72, 192.168.2.73], zone2.host_ids=[6], host_ids=[5, 6, 7], zone1.host_ids=[5], zone_names=[zone1, zone2, zone3]})

2021-03-22 20:26:05.462  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.c.metadb.task.TaskInstanceEntity   : executor node ip:192.168.2.70

2021-03-22 20:26:05.623  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=root, cmd=ifconfig  2> /dev/null | grep '192.168.2.71' -B 1 | grep -E 'flags|HWaddr' | awk -F '[: ]+' '{print $1}'

2021-03-22 20:26:05.744  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=ens192

2021-03-22 20:26:05.786  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=admin, cmd=cd /home/admin/oceanbase; ulimit -s 10240; ulimit -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD='' /home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone1 -d /home/admin/oceanbase/store/sandata -l info -o'obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648'

2021-03-22 20:26:07.484  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=/home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone1 -d /home/admin/oceanbase/store/sandata -l info -oobconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648
devname: ens192
mysql port: 2881
rpc port: 2882
appname: sandata
zone: zone1
data_dir: /home/admin/oceanbase/store/sandata
log level: info
optstr: obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648

2021-03-22 20:26:07.584  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=root, cmd=ifconfig  2> /dev/null | grep '192.168.2.72' -B 1 | grep -E 'flags|HWaddr' | awk -F '[: ]+' '{print $1}'

2021-03-22 20:26:07.622  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=ens192

2021-03-22 20:26:07.649  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=admin, cmd=cd /home/admin/oceanbase; ulimit -s 10240; ulimit -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD='' /home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone2 -d /home/admin/oceanbase/store/sandata -l info -o'obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648'

2021-03-22 20:26:07.847  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=/home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone2 -d /home/admin/oceanbase/store/sandata -l info -oobconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648
devname: ens192
mysql port: 2881
rpc port: 2882
appname: sandata
zone: zone2
data_dir: /home/admin/oceanbase/store/sandata
log level: info
optstr: obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648

2021-03-22 20:26:07.933  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=root, cmd=ifconfig  2> /dev/null | grep '192.168.2.73' -B 1 | grep -E 'flags|HWaddr' | awk -F '[: ]+' '{print $1}'

2021-03-22 20:26:07.984  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=ens192

2021-03-22 20:26:08.027  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=admin, cmd=cd /home/admin/oceanbase; ulimit -s 10240; ulimit -c unlimited; LD_LIBRARY_PATH=/home/admin/oceanbase/lib:$LD_LIBRARY_PATH LD_PRELOAD='' /home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone3 -d /home/admin/oceanbase/store/sandata -l info -o'obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648'

2021-03-22 20:26:08.378  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=/home/admin/oceanbase/bin/observer -i ens192 -pxxx 2881 -P 2882 -n sandata -z zone3 -d /home/admin/oceanbase/store/sandata -l info -oobconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648
devname: ens192
mysql port: 2881
rpc port: 2882
appname: sandata
zone: zone3
data_dir: /home/admin/oceanbase/store/sandata
log level: info
optstr: obconfig_url=http://192.168.2.70:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=sandata,rootservice_list=192.168.2.71:2882:2881;192.168.2.72:2882:2881;192.168.2.73:2882:2881,datafile_disk_percentage=95,config_additional_dir=/data/log1/sandata/etc2,/data/1/sandata/etc3,cluster_id=1,cpu_count=32,memory_limit=15G,system_memory=5G,cache_wash_threshold=1GB,net_thread_count=4,workers_per_cpu_quota=2,stack_size=512K,__min_full_resource_pool_memory=2147483648

2021-03-22 20:26:08.408  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.s.t.b.c.helper.ObServerTaskHelper  : begin to check server accessible, hostIds=[5, 6, 7]

2021-03-22 20:26:08.456  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot  -e "select 1 from dual;"

2021-03-22 20:26:08.789 ERROR 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : call pos method failed, server IP: 192.168.2.71, port: 62888, args: admin mysql -h127.1 -P2881 -uroot  -e "select 1 from dual;", code: 1, result: ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111)

2021-03-22 20:26:08.819  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot  -e "select 1 from dual;"

2021-03-22 20:26:08.928 ERROR 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : call pos method failed, server IP: 192.168.2.72, port: 62888, args: admin mysql -h127.1 -P2881 -uroot  -e "select 1 from dual;", code: 1, result: ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111)

2021-03-22 20:26:08.972  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot  -e "select 1 from dual;"

2021-03-22 20:26:09.211 ERROR 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : call pos method failed, server IP: 192.168.2.73, port: 62888, args: admin mysql -h127.1 -P2881 -uroot  -e "select 1 from dual;", code: 1, result: ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111)

2021-03-22 20:26:09.225  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.s.t.b.c.helper.ObServerTaskHelper  : there exists server(s) still not accessible, hostIds=5,6,7

2021-03-22 20:26:09.262  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] com.alipay.ocp.common.pattern.Retry      : wait for 30 seconds

2021-03-22 20:26:39.343  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot  -e "select 1 from dual;"

2021-03-22 20:26:40.734  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=1
1

2021-03-22 20:26:40.778  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.72, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot  -e "select 1 from dual;"

2021-03-22 20:26:42.477  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=1
1

2021-03-22 20:26:42.555  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.73, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot  -e "select 1 from dual;"

2021-03-22 20:26:44.032  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] result=1
1

2021-03-22 20:26:44.045  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.s.t.b.c.helper.ObServerTaskHelper  : all ob server accessible

2021-03-22 20:26:44.055  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.s.t.b.cluster.BootStrapObTask      : bootstrap command: mysql -h127.1 -P2881 -uroot -e "set ob_query_timeout=600000000; ALTER SYSTEM BOOTSTRAP  REGION 'beijing' ZONE 'zone1' SERVER '192.168.2.71:2882', REGION 'beijing' ZONE 'zone2' SERVER '192.168.2.72:2882', REGION 'beijing' ZONE 'zone3' SERVER '192.168.2.73:2882' ;"

2021-03-22 20:26:44.088  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : [OcpAgentUtils.runCmd] svrIp=192.168.2.71, port=62888, user=admin, cmd=mysql -h127.1 -P2881 -uroot -e "set ob_query_timeout=600000000; ALTER SYSTEM BOOTSTRAP  REGION 'beijing' ZONE 'zone1' SERVER '192.168.2.71:2882', REGION 'beijing' ZONE 'zone2' SERVER '192.168.2.72:2882', REGION 'beijing' ZONE 'zone3' SERVER '192.168.2.73:2882' ;"

2021-03-22 20:36:44.603 ERROR 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.ocp.core.task.util.OcpAgentUtils     : call pos method failed, server IP: 192.168.2.71, port: 62888, args: admin mysql -h127.1 -P2881 -uroot -e "set ob_query_timeout=600000000; ALTER SYSTEM BOOTSTRAP  REGION 'beijing' ZONE 'zone1' SERVER '192.168.2.71:2882', REGION 'beijing' ZONE 'zone2' SERVER '192.168.2.72:2882', REGION 'beijing' ZONE 'zone3' SERVER '192.168.2.73:2882' ;", code: 1, result: ERROR 4012 (HY000) at line 1: Timeout

2021-03-22 20:36:44.648 ERROR 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.core.task.manager.TaskManagerImpl  : got exception while execute task:

com.alipay.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=COMMON_RPC_EXECUTE_FAILED, args=192.168.2.71,62888,admin mysql -h127.1 -P2881 -uroot -e "set ob_query_timeout=600000000; ALTER SYSTEM BOOTSTRAP  REGION 'beijing' ZONE 'zone1' SERVER '192.168.2.71:2882', REGION 'beijing' ZONE 'zone2' SERVER '192.168.2.72:2882', REGION 'beijing' ZONE 'zone3' SERVER '192.168.2.73:2882' ;",1,ERROR 4012 (HY000) at line 1: Timeout
	at com.alipay.ocp.core.task.util.OcpAgentUtils.executePosMethod(OcpAgentUtils.java:265) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
	at com.alipay.ocp.core.task.util.OcpAgentUtils.callPosMethod(OcpAgentUtils.java:128) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
	at com.alipay.ocp.core.task.util.OcpAgentUtils.runCmd(OcpAgentUtils.java:50) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
	at com.alipay.ocp.service.compute.host.HostManager.executeRpcCommand(HostManager.java:82) ~[ocp-service-2.5.1-20210220.jar!/:2.5.1-20210220]
	at com.alipay.ocp.service.task.business.cluster.BootStrapObTask.run(BootStrapObTask.java:112) ~[ocp-service-2.5.1-20210220.jar!/:2.5.1-20210220]
	at com.alipay.ocp.core.metadb.task.TaskInstanceEntity.run(TaskInstanceEntity.java:192) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
	at com.alipay.ocp.core.task.manager.TaskManagerImpl.lambda$null$1(TaskManagerImpl.java:186) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
	at com.alipay.ocp.core.task.manager.TaskManagerImpl.redirectOutputIfNotSysSchedule(TaskManagerImpl.java:241) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
	at com.alipay.ocp.core.task.manager.TaskManagerImpl.lambda$executeTask$2(TaskManagerImpl.java:159) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_275]
	at java.util.concurrent.FutureTask.run(F
utureTask.java:266) ~[na:1.8.0_275]
	at com.alipay.ocp.core.thread.TraceDecorator.lambda$decorate$0(TraceDecorator.java:28) ~[ocp-core-2.5.1-20210220.jar!/:2.5.1-20210220]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_275]
	at java.lang.Thread.run(Thread.java:874) ~[na:1.8.0_275]


2021-03-22 20:36:44.684  INFO 53 --- [operation-task-6,0b46837642d7498b,e8dca7a420d5] c.a.o.c.metadb.task.TaskInstanceEntity   : set state failed for task: 1833

 

【问题分析】

我们可以在observer上面测试一下相关命令是否可以正常执行:

[root@observer1 clonescripts]#  mysql -h127.1 -P2881 -uroot  -e "select 1 from dual"
+---+
| 1 |
+---+
| 1 |
+---+

 

[root@observer1 clonescripts]# mysql -h127.1 -P2881 -uroot
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 3221315916
Server version: 5.7.25 OceanBase 2.2.75 (r20210108170301-c8c037dd6352fa0d00bd8a7105a1b566a5bc790d) (Built Jan  8 2021 17:31:33)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> ALTER SYSTEM BOOTSTRAP  REGION 'beijing' ZONE 'zone1' SERVER '192.168.2.71:2882', REGION 'beijing' ZONE 'zone2' SERVER '192.168.2.72:2882', REGION 'beijing' ZONE 'zone3' SERVER '192.168.2.73:2882' ;
ERROR 4015 (HY000): System error
MySQL [(none)]> 

如果出现上述错误,我采用的方法是删除集群,删除obsever主机,重新配置来解决。

发表回复
你还没有登录,请先 登录或 注册!